search

go-graphite / go-carbon

Golang implementation of Graphite/Carbon server with classic architecture: Agent -> Cache -> Persister
MIT License
805 stars 123 forks source link

No data in query response #159

Closed con5cience closed 7 years ago

con5cience commented 7 years ago

In AWS, using Grafana 4.2.0 with Go-Carbon 1.9.1, talking to Carbonapi, which is talking to embedded Carbonserver in Go-Carbon.

I can use whisper-fetch.py on the Carbon server itself and there is data present. If I try to use curl, it appears to transact successfully (exit 0) but doesn't return any data. In Grafana I get no datapoints and the Developer console throws an error indicating there was no data attribute in the response.

Things I've tried: Different versions of Go-Carbon, opening up Inbound/Outbound ACLs on the security groups to permit all traffic. Routing tables are simple, 0.0.0.0/0 to the Internet and local for everything else on the subnet. Tcpdump shows the packets coming in, strace shows the calls writing metrics to disk. I don't seem to be able to debug queries otherwise.

Nothing useful in any logfile anywhere even with raised loglevels -- the carbonapi logs simply correlate that there was 0 response data sent.

Thanks in advance for any help.

con5cience commented 7 years ago 
# curl --verbose "http://172.22.1.175:8082/render?target=monitoring.Staging.graphite01.memory.free&format=json"
* About to connect() to 172.22.1.175 port 8082 (#0)
*   Trying 172.22.1.175...
* Connected to 172.22.1.175 (172.22.1.175) port 8082 (#0)
> GET /render?target=monitoring.Staging.graphite01.memory.free&format=json HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 172.22.1.175:8082
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Thu, 20 Apr 2017 20:02:40 GMT
< Content-Length: 0
< 
* Connection #0 to host 172.22.1.175 left intact
Civil commented 7 years ago

Can you please post the logs on debug level (at least carbonserver module set on debug level) and a config you are using?

con5cience commented 7 years ago

Configs:

[common]
user = ""
logfile = "/var/log/go-carbon/go-carbon.log"
log-level = "debug"
graph-prefix = "carbon.agents.{host}"
max-cpu = 2
metric-interval = "0m10s"
metric-endpoint = "local"

[whisper]
data-dir = "/mnt/data3/storage/whisper"
schemas-file = "/etc/go-carbon/storage-schemas.conf"
aggregation-file = "/etc/go-carbon/storage-aggregation.conf"
workers = 1
max-updates-per-second = 0
enabled = true
sparse-create = false

[cache]
max-size = 1000000
input-buffer = 51200
write-strategy = "noop"

[udp]
listen = ":2003"
enabled = true
log-incomplete = false

[tcp]
listen = ":2003"
enabled = true

[pickle]
listen = ":2004"
max-message-size = 67108864
enabled = true

[carbonlink]
listen = ":7002"
enabled = true
read-timeout = "30s"
query-timeout = "100ms"

[carbonserver]
listen = ":8080"
enabled = true
buckets = 10
max-globs = 100
metrics-as-counters = true
read-timeout = "1m0s"
write-timeout = "1m0s"
scan-frequency = "5m0s"
query-cache-enabled = true
query-cache-size-mb = 0
find-cache-enabled = true

[dump]
enabled = false
path = ""
restore-per-second = 0

[pprof]
listen = "localhost:7007"
enabled = false
con5cience commented 7 years ago

Log sample, this loops over and over again:

[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.runtime.GOMAXPROCS=2
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.runtime.NumGoroutine=23
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.cache.size=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.cache.metrics=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.cache.maxSize=1e+06
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.cache.queries=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.cache.overflow=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.cache.queueBuildCount=100
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.cache.queueBuildTimeMs=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.cache.queueWriteoutTime=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.cache.notConfirmedUsedTotal=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.cache.notConfirmedLenTotal=4096
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.cache.notConfirmedShardUsedMax=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.cache.notConfirmedShardLenMax=4
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.render_requests=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.render_errors=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.notfound=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.find_requests=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.find_errors=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.find_zero=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.cache_hit=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.cache_miss=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.cache_work_time_ns=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.cache_wait_time_fetch_ns=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.cache_requests=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.disk_wait_time_ns=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.disk_requests=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.points_returned=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.metrics_returned=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.alloc=2.647896e+06
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.total_alloc=8.391688e+06
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.num_gc=2
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.pause_ns=260889
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.requests_in_0ms_to_100ms=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.requests_in_100ms_to_200ms=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.requests_in_200ms_to_300ms=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.requests_in_300ms_to_400ms=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.requests_in_400ms_to_500ms=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.requests_in_500ms_to_600ms=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.requests_in_600ms_to_700ms=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.requests_in_700ms_to_800ms=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.requests_in_800ms_to_900ms=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.requests_in_900ms_to_1000ms=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.carbonserver.requests_in_1000ms_to_1100ms=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.udp.metricsReceived=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.udp.incompleteReceived=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.udp.errors=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.tcp.metricsReceived=67
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.tcp.active=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.tcp.errors=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.pickle.metricsReceived=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.pickle.active=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.pickle.errors=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.persister.updateOperations=126
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.persister.committedPoints=126
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.persister.pointsPerUpdate=1
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.persister.created=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.persister.maxUpdatesPerSecond=0
[2017-04-20 20:31:57] I [stat] carbon.agents.graphite01.persister.workers=1
[2017-04-20 20:31:57] D nextRebuildOnce.Do: "2017-04-20 20:31:57.43618374 +0000 UTC" "2017-04-20 20:31:57.535051774 +0000 UTC"
[2017-04-20 20:31:57] D sleep 98.868034ms before rebuild
[2017-04-20 20:31:57] D nextRebuildOnce.Do: "2017-04-20 20:31:57.535426155 +0000 UTC" "2017-04-20 20:31:57.635409803 +0000 UTC"
[2017-04-20 20:31:57] D sleep 99.983648ms before rebuild
[2017-04-20 20:31:57] D nextRebuildOnce.Do: "2017-04-20 20:31:57.635787667 +0000 UTC" "2017-04-20 20:31:57.735782024 +0000 UTC"
[2017-04-20 20:31:57] D sleep 99.994357ms before rebuild
[2017-04-20 20:31:57] D nextRebuildOnce.Do: "2017-04-20 20:31:57.736157819 +0000 UTC" "2017-04-20 20:31:57.836151729 +0000 UTC"
[2017-04-20 20:31:57] D sleep 99.99391ms before rebuild
[2017-04-20 20:31:57] D nextRebuildOnce.Do: "2017-04-20 20:31:57.836528017 +0000 UTC" "2017-04-20 20:31:57.936521385 +0000 UTC"
[2017-04-20 20:31:57] D sleep 99.993368ms before rebuild
[2017-04-20 20:31:57] D nextRebuildOnce.Do: "2017-04-20 20:31:57.936906262 +0000 UTC" "2017-04-20 20:31:58.036896851 +0000 UTC"
[2017-04-20 20:31:57] D sleep 99.990589ms before rebuild
[2017-04-20 20:31:58] D nextRebuildOnce.Do: "2017-04-20 20:31:58.03820151 +0000 UTC" "2017-04-20 20:31:58.13819484 +0000 UTC"
[2017-04-20 20:31:58] D sleep 99.99333ms before rebuild
[2017-04-20 20:31:58] D nextRebuildOnce.Do: "2017-04-20 20:31:58.138610484 +0000 UTC" "2017-04-20 20:31:58.238604441 +0000 UTC"
[2017-04-20 20:31:58] D sleep 99.993957ms before rebuild
[2017-04-20 20:31:58] D nextRebuildOnce.Do: "2017-04-20 20:31:58.239022959 +0000 UTC" "2017-04-20 20:31:58.339016279 +0000 UTC"
[2017-04-20 20:31:58] D sleep 99.99332ms before rebuild
[2017-04-20 20:31:58] D nextRebuildOnce.Do: "2017-04-20 20:31:58.339383031 +0000 UTC" "2017-04-20 20:31:58.439376261 +0000 UTC"
[2017-04-20 20:31:58] D sleep 99.99323ms before rebuild
[2017-04-20 20:31:58] D nextRebuildOnce.Do: "2017-04-20 20:31:58.439882491 +0000 UTC" "2017-04-20 20:31:58.539876292 +0000 UTC"
[2017-04-20 20:31:58] D sleep 99.993801ms before rebuild
[2017-04-20 20:31:58] D nextRebuildOnce.Do: "2017-04-20 20:31:58.540260889 +0000 UTC" "2017-04-20 20:31:58.640254702 +0000 UTC"
[2017-04-20 20:31:58] D sleep 99.993813ms before rebuild
[2017-04-20 20:31:58] D nextRebuildOnce.Do: "2017-04-20 20:31:58.640626133 +0000 UTC" "2017-04-20 20:31:58.74062013 +0000 UTC"
[2017-04-20 20:31:58] D sleep 99.993997ms before rebuild
[2017-04-20 20:31:58] D nextRebuildOnce.Do: "2017-04-20 20:31:58.740988042 +0000 UTC" "2017-04-20 20:31:58.840981848 +0000 UTC"
[2017-04-20 20:31:58] D sleep 99.993806ms before rebuild
[2017-04-20 20:31:58] D nextRebuildOnce.Do: "2017-04-20 20:31:58.841423652 +0000 UTC" "2017-04-20 20:31:58.941417095 +0000 UTC"
[2017-04-20 20:31:58] D sleep 99.993443ms before rebuild
[2017-04-20 20:31:58] D nextRebuildOnce.Do: "2017-04-20 20:31:58.942006273 +0000 UTC" "2017-04-20 20:31:59.041999171 +0000 UTC"
[2017-04-20 20:31:58] D sleep 99.992898ms before rebuild
[2017-04-20 20:31:59] D nextRebuildOnce.Do: "2017-04-20 20:31:59.042390089 +0000 UTC" "2017-04-20 20:31:59.142383146 +0000 UTC"
[2017-04-20 20:31:59] D sleep 99.993057ms before rebuild
[2017-04-20 20:31:59] D nextRebuildOnce.Do: "2017-04-20 20:31:59.14280187 +0000 UTC" "2017-04-20 20:31:59.242795911 +0000 UTC"
[2017-04-20 20:31:59] D sleep 99.994041ms before rebuild
[2017-04-20 20:31:59] D nextRebuildOnce.Do: "2017-04-20 20:31:59.24318688 +0000 UTC" "2017-04-20 20:31:59.343179039 +0000 UTC"
[2017-04-20 20:31:59] D sleep 99.992159ms before rebuild
[2017-04-20 20:31:59] D nextRebuildOnce.Do: "2017-04-20 20:31:59.343578077 +0000 UTC" "2017-04-20 20:31:59.443571827 +0000 UTC"
[2017-04-20 20:31:59] D sleep 99.99375ms before rebuild
[2017-04-20 20:31:59] D nextRebuildOnce.Do: "2017-04-20 20:31:59.443966242 +0000 UTC" "2017-04-20 20:31:59.543959746 +0000 UTC"
[2017-04-20 20:31:59] D sleep 99.993504ms before rebuild
[2017-04-20 20:31:59] D nextRebuildOnce.Do: "2017-04-20 20:31:59.544539979 +0000 UTC" "2017-04-20 20:31:59.644529172 +0000 UTC"
[2017-04-20 20:31:59] D sleep 99.989193ms before rebuild
[2017-04-20 20:31:59] D nextRebuildOnce.Do: "2017-04-20 20:31:59.645002012 +0000 UTC" "2017-04-20 20:31:59.744995702 +0000 UTC"
[2017-04-20 20:31:59] D sleep 99.99369ms before rebuild
[2017-04-20 20:31:59] D nextRebuildOnce.Do: "2017-04-20 20:31:59.745385907 +0000 UTC" "2017-04-20 20:31:59.845379387 +0000 UTC"
[2017-04-20 20:31:59] D sleep 99.99348ms before rebuild
[2017-04-20 20:31:59] D nextRebuildOnce.Do: "2017-04-20 20:31:59.845765175 +0000 UTC" "2017-04-20 20:31:59.945758335 +0000 UTC"
[2017-04-20 20:31:59] D sleep 99.99316ms before rebuild
[2017-04-20 20:31:59] D nextRebuildOnce.Do: "2017-04-20 20:31:59.946170877 +0000 UTC" "2017-04-20 20:32:00.046164262 +0000 UTC"
[2017-04-20 20:31:59] D sleep 99.993385ms before rebuild
[2017-04-20 20:32:00] D nextRebuildOnce.Do: "2017-04-20 20:32:00.046544835 +0000 UTC" "2017-04-20 20:32:00.146539006 +0000 UTC"
[2017-04-20 20:32:00] D sleep 99.994171ms before rebuild
[2017-04-20 20:32:00] D nextRebuildOnce.Do: "2017-04-20 20:32:00.146917547 +0000 UTC" "2017-04-20 20:32:00.246911504 +0000 UTC"
[2017-04-20 20:32:00] D sleep 99.993957ms before rebuild
[2017-04-20 20:32:00] D nextRebuildOnce.Do: "2017-04-20 20:32:00.24730676 +0000 UTC" "2017-04-20 20:32:00.347300855 +0000 UTC"
[2017-04-20 20:32:00] D sleep 99.994095ms before rebuild
[2017-04-20 20:32:00] D nextRebuildOnce.Do: "2017-04-20 20:32:00.347721351 +0000 UTC" "2017-04-20 20:32:00.447698394 +0000 UTC"
[2017-04-20 20:32:00] D sleep 99.977043ms before rebuild
[2017-04-20 20:32:00] D nextRebuildOnce.Do: "2017-04-20 20:32:00.448093666 +0000 UTC" "2017-04-20 20:32:00.548087206 +0000 UTC"
[2017-04-20 20:32:00] D sleep 99.99354ms before rebuild
[2017-04-20 20:32:00] D nextRebuildOnce.Do: "2017-04-20 20:32:00.548486894 +0000 UTC" "2017-04-20 20:32:00.648480771 +0000 UTC"
[2017-04-20 20:32:00] D sleep 99.993877ms before rebuild
[2017-04-20 20:32:00] D nextRebuildOnce.Do: "2017-04-20 20:32:00.648851815 +0000 UTC" "2017-04-20 20:32:00.748845348 +0000 UTC"
[2017-04-20 20:32:00] D sleep 99.993533ms before rebuild
[2017-04-20 20:32:00] D nextRebuildOnce.Do: "2017-04-20 20:32:00.750843347 +0000 UTC" "2017-04-20 20:32:00.849236604 +0000 UTC"
[2017-04-20 20:32:00] D sleep 98.393257ms before rebuild
[2017-04-20 20:32:00] D nextRebuildOnce.Do: "2017-04-20 20:32:00.849594756 +0000 UTC" "2017-04-20 20:32:00.949588071 +0000 UTC"
[2017-04-20 20:32:00] D sleep 99.993315ms before rebuild
[2017-04-20 20:32:00] D nextRebuildOnce.Do: "2017-04-20 20:32:00.949992625 +0000 UTC" "2017-04-20 20:32:01.049987109 +0000 UTC"
[2017-04-20 20:32:00] D sleep 99.994484ms before rebuild
[2017-04-20 20:32:01] D nextRebuildOnce.Do: "2017-04-20 20:32:01.050387013 +0000 UTC" "2017-04-20 20:32:01.150380496 +0000 UTC"
[2017-04-20 20:32:01] D sleep 99.993483ms before rebuild
[2017-04-20 20:32:01] D nextRebuildOnce.Do: "2017-04-20 20:32:01.150780619 +0000 UTC" "2017-04-20 20:32:01.250773705 +0000 UTC"
[2017-04-20 20:32:01] D sleep 99.993086ms before rebuild
[2017-04-20 20:32:01] D nextRebuildOnce.Do: "2017-04-20 20:32:01.251164968 +0000 UTC" "2017-04-20 20:32:01.351158671 +0000 UTC"
[2017-04-20 20:32:01] D sleep 99.993703ms before rebuild
[2017-04-20 20:32:01] D nextRebuildOnce.Do: "2017-04-20 20:32:01.351541992 +0000 UTC" "2017-04-20 20:32:01.451535236 +0000 UTC"
[2017-04-20 20:32:01] D sleep 99.993244ms before rebuild
[2017-04-20 20:32:01] D nextRebuildOnce.Do: "2017-04-20 20:32:01.451919944 +0000 UTC" "2017-04-20 20:32:01.551913536 +0000 UTC"
[2017-04-20 20:32:01] D sleep 99.993592ms before rebuild
[2017-04-20 20:32:01] D nextRebuildOnce.Do: "2017-04-20 20:32:01.552315401 +0000 UTC" "2017-04-20 20:32:01.652309023 +0000 UTC"
[2017-04-20 20:32:01] D sleep 99.993622ms before rebuild
[2017-04-20 20:32:01] D nextRebuildOnce.Do: "2017-04-20 20:32:01.652701271 +0000 UTC" "2017-04-20 20:32:01.752695173 +0000 UTC"
[2017-04-20 20:32:01] D sleep 99.993902ms before rebuild
[2017-04-20 20:32:01] D nextRebuildOnce.Do: "2017-04-20 20:32:01.753089197 +0000 UTC" "2017-04-20 20:32:01.853083477 +0000 UTC"
[2017-04-20 20:32:01] D sleep 99.99428ms before rebuild
[2017-04-20 20:32:01] D nextRebuildOnce.Do: "2017-04-20 20:32:01.853489343 +0000 UTC" "2017-04-20 20:32:01.95348327 +0000 UTC"
[2017-04-20 20:32:01] D sleep 99.993927ms before rebuild
[2017-04-20 20:32:01] D nextRebuildOnce.Do: "2017-04-20 20:32:01.953861331 +0000 UTC" "2017-04-20 20:32:02.053855175 +0000 UTC"
[2017-04-20 20:32:01] D sleep 99.993844ms before rebuild
[2017-04-20 20:32:02] D nextRebuildOnce.Do: "2017-04-20 20:32:02.054246732 +0000 UTC" "2017-04-20 20:32:02.154240615 +0000 UTC"
[2017-04-20 20:32:02] D sleep 99.993883ms before rebuild
[2017-04-20 20:32:02] D nextRebuildOnce.Do: "2017-04-20 20:32:02.154605141 +0000 UTC" "2017-04-20 20:32:02.254598253 +0000 UTC"
[2017-04-20 20:32:02] D sleep 99.993112ms before rebuild
[2017-04-20 20:32:02] D nextRebuildOnce.Do: "2017-04-20 20:32:02.255001857 +0000 UTC" "2017-04-20 20:32:02.354995033 +0000 UTC"
[2017-04-20 20:32:02] D sleep 99.993176ms before rebuild
[2017-04-20 20:32:02] D nextRebuildOnce.Do: "2017-04-20 20:32:02.355387097 +0000 UTC" "2017-04-20 20:32:02.455380947 +0000 UTC"
[2017-04-20 20:32:02] D sleep 99.99385ms before rebuild
[2017-04-20 20:32:02] D nextRebuildOnce.Do: "2017-04-20 20:32:02.455715868 +0000 UTC" "2017-04-20 20:32:02.555709029 +0000 UTC"
[2017-04-20 20:32:02] D sleep 99.993161ms before rebuild
[2017-04-20 20:32:02] D nextRebuildOnce.Do: "2017-04-20 20:32:02.556145055 +0000 UTC" "2017-04-20 20:32:02.656139325 +0000 UTC"
[2017-04-20 20:32:02] D sleep 99.99427ms before rebuild
[2017-04-20 20:32:02] D nextRebuildOnce.Do: "2017-04-20 20:32:02.656500945 +0000 UTC" "2017-04-20 20:32:02.756494576 +0000 UTC"
[2017-04-20 20:32:02] D sleep 99.993631ms before rebuild
[2017-04-20 20:32:02] D nextRebuildOnce.Do: "2017-04-20 20:32:02.756903093 +0000 UTC" "2017-04-20 20:32:02.856896634 +0000 UTC"
[2017-04-20 20:32:02] D sleep 99.993541ms before rebuild
[2017-04-20 20:32:02] D nextRebuildOnce.Do: "2017-04-20 20:32:02.857276471 +0000 UTC" "2017-04-20 20:32:02.957270037 +0000 UTC"
[2017-04-20 20:32:02] D sleep 99.993566ms before rebuild
[2017-04-20 20:32:02] D nextRebuildOnce.Do: "2017-04-20 20:32:02.957649509 +0000 UTC" "2017-04-20 20:32:03.057633888 +0000 UTC"
[2017-04-20 20:32:02] D sleep 99.984379ms before rebuild
[2017-04-20 20:32:03] D nextRebuildOnce.Do: "2017-04-20 20:32:03.059014607 +0000 UTC" "2017-04-20 20:32:03.159008755 +0000 UTC"
[2017-04-20 20:32:03] D sleep 99.994148ms before rebuild
[2017-04-20 20:32:03] D nextRebuildOnce.Do: "2017-04-20 20:32:03.159415266 +0000 UTC" "2017-04-20 20:32:03.259409032 +0000 UTC"
[2017-04-20 20:32:03] D sleep 99.993766ms before rebuild
[2017-04-20 20:32:03] D nextRebuildOnce.Do: "2017-04-20 20:32:03.259779287 +0000 UTC" "2017-04-20 20:32:03.359772642 +0000 UTC"
[2017-04-20 20:32:03] D sleep 99.993355ms before rebuild
[2017-04-20 20:32:03] D nextRebuildOnce.Do: "2017-04-20 20:32:03.360177121 +0000 UTC" "2017-04-20 20:32:03.460171076 +0000 UTC"
[2017-04-20 20:32:03] D sleep 99.993955ms before rebuild
[2017-04-20 20:32:03] D nextRebuildOnce.Do: "2017-04-20 20:32:03.460540083 +0000 UTC" "2017-04-20 20:32:03.560533833 +0000 UTC"
[2017-04-20 20:32:03] D sleep 99.99375ms before rebuild
[2017-04-20 20:32:03] D nextRebuildOnce.Do: "2017-04-20 20:32:03.56094354 +0000 UTC" "2017-04-20 20:32:03.660937456 +0000 UTC"
[2017-04-20 20:32:03] D sleep 99.993916ms before rebuild
[2017-04-20 20:32:03] D nextRebuildOnce.Do: "2017-04-20 20:32:03.66132856 +0000 UTC" "2017-04-20 20:32:03.761322881 +0000 UTC"
Civil commented 7 years ago

8082 - port of carbonapi, right? What you'll get in the result if you'll do the same query but again go-carbon (port 8080)?

con5cience commented 7 years ago 
[root@grafana01 ~]# curl --verbose "http://172.22.1.175:8080/render?target=monitoring.Staging.graphite01.memory.free&format=json"
* About to connect() to 172.22.1.175 port 8080 (#0)
*   Trying 172.22.1.175...
* Connected to 172.22.1.175 (172.22.1.175) port 8080 (#0)
> GET /render?target=monitoring.Staging.graphite01.memory.free&format=json HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 172.22.1.175:8080
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Location: /render/
< Vary: Accept-Encoding
< Date: Thu, 20 Apr 2017 21:23:44 GMT
< Content-Length: 43
< Content-Type: text/html; charset=utf-8
< 
<a href="/render/">Moved Permanently</a>.

* Connection #0 to host 172.22.1.175 left intact
con5cience commented 7 years ago

Then when I add the slash:

[root@grafana01 ~]# curl --verbose "http://172.22.1.175:8080/render/?target=monitoring.Staging.graphite01.memory.free&format=json&from=-1w&until=now"
* About to connect() to 172.22.1.175 port 8080 (#0)
*   Trying 172.22.1.175...
* Connected to 172.22.1.175 (172.22.1.175) port 8080 (#0)
> GET /render/?target=monitoring.Staging.graphite01.memory.free&format=json&from=-1w&until=now HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 172.22.1.175:8080
> Accept: */*
> 
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< Vary: Accept-Encoding
< X-Content-Type-Options: nosniff
< Date: Thu, 20 Apr 2017 21:26:08 GMT
< Content-Length: 38
< 
Bad request (invalid from/until time)
* Connection #0 to host 172.22.1.175 left intact

Same result if I don't override from/to/until.

Civil commented 7 years ago

go-carbon's carbonserver always require from and until, because this is the way how graphite-web do cluster servers. Also from and until must be unix timestamps.

For me it seems that you have a problem with carbonapi, not go-carbon.

con5cience commented 7 years ago

Querying :8080 with epoch timestamps does seem to work.

Also at second glance it looks like there's a bit of a trainwreck with carbonzipper and support for various versions of protobuf/carbonserver/carbonapi/etc.

con5cience commented 7 years ago

Thank you for your help!

Civil commented 7 years ago

For carbonserver - it supports only the subset of graphite-web that's used by graphtie-web clustering in the same way. That's why it requires from + until and requires them to be unix timestamps.

Also at second glance it looks like there's a bit of a trainwreck with carbonzipper and support for various versions of protobuf/carbonserver/carbonapi/etc.

Yeah, there were some problems with that. carbonapi in master works ok with both protobuf2 and 3, though carbonapi 0.7 requires go-carbon that could answer with format=protobuf3, which is currently "master" branch.