search

go-graphite / carbonzipper

proxy to transparently merge graphite carbon backends
Other
104 stars 29 forks source link

carbonzipper 0.74 not returning the results for the multiple queries. #72

Closed dreddynarahari closed 5 years ago

dreddynarahari commented 5 years ago

I have two queries below target=local.monitoring.U.rg.adc01.servers.ossgmsdataadc0101.cache.notConfirmedShardLenMax&target=local.monitoring.U.rg.adc01.servers.ossgmsdataadc0101.cache.maxSize But zipper returns values only for the first part of the query.

2018-09-27T16:31:25.838-0500 DEBUG render got render request {"memory_usage_bytes": 0, "handler": "render", "carbonzipper_uuid": "2bcfb82f-2257-4543-a284-06813b3e45cb", "carbonapi_uuid": "", "request": "/render/?format=pickle&local=1&noCache=1&from=1538083585&until=1538083885&target=local.monitoring.U.rg.adc01.servers.ossgmsdataadc0101.cache.notConfirmedShardLenMax&target=local.monitoring.U.rg.adc01.servers.ossgmsdataadc0101.cache.maxSize&now=1538083885"} 2018-09-27T16:31:25.838-0500 DEBUG render querying servers {"memory_usage_bytes": 0, "handler": "render", "carbonzipper_uuid": "2bcfb82f-2257-4543-a284-06813b3e45cb", "carbonapi_uuid": "", "handler": "multiGet", "servers": ["http://172.16.0.162:8080", "http://172.16.0.163:8080"], "uri": "/render/?format=protobuf&from=1538083585&target=ocal.monitoring.U.rg.adc01.servers.ossgmsdataadc0101.cache.notConfirmedShardLenMax&until=1538083885"} 2018-09-27T16:31:26.518-0500 ERROR render query error {"memory_usage_bytes": 0, "handler": "render", "carbonzipper_uuid": "2bcfb82f-2257-4543-a284-06813b3e45cb", "carbonapi_uuid": "", "handler": "multiGet", "handler": "singleGet", "query": "http://172.16.0.163:8080//render/?format=protobuf&from=1538083585&target=ocal.monitoring.U.rg.adc01.servers.ossgmsdataadc0101.cache.notConfirmedShardLenMax&until=1538083885", "error": "Get http://172.16.0.163:8080/render/?format=protobuf&from=1538083585&target=local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101a.cache.notConfirmedShardLenMax&until=1538083885: dial tcp 172.16.0.163:8080: getsockopt: no route to host"} 2018-09-27T16:31:26.518-0500 DEBUG zipper_render decoded response {"name": "ocal.monitoring.U.rg.adc01.servers.ossgmsdataadc0101.cache.notConfirmedShardLenMax", "decoded": [{"name":"ocal.monitoring.U.rg.adc01.servers.ossgmsdataadc0101.cache.notConfirmedShardLenMax","startTime":1538083800,"stopTime":1538084100,"stepTime":300,"values":[4],"isAbsent":[false]}]} 2018-09-27T16:31:26.518-0500 DEBUG zipper_render only one decoded response to merge {"name": "local.monitoring.U.rg.adc01.servers.ossgmsdataadc0101.cache.notConfirmedShardLenMax"} 2018-09-27T16:31:26.518-0500 INFO access request served {"handler": "render", "carbonzipper_uuid": "2bcfb82f-2257-4543-a284-06813b3e45cb", "carbonapi_uuid": "", "format": "pickle", "target": "local.monitoring.U.rg.adc01.servers.ossgmsdataadc0101.cache.notConfirmedShardLenMax", "memory_usage_bytes": 0, "http_code": 200, "runtime_seconds": 0.67930205}

Civil commented 5 years ago
  1. getsockopt: no route to host" means networking issue - your carbonzipper server cannot reach backends.
  2. carbonzipper 0.7x doesn't support multiple targets in the request. As far as I remember this is supported only in current master.
dreddynarahari commented 5 years ago

Retried the query after removing the faulty server in my local DEV env: 2018-09-27T17:00:28.996-0500 DEBUG render got render request {"memory_usage_bytes": 0, "handler": "render", "carbonzipper_uuid": "7514f628-e9ec-41f3-b287-dcadf244e7db", "carbonapi_uuid": "", "request": "/render/?format=pickle&local=1&noCache=1&from=1538085328&until=1538085628&target=xxx.cache.notConfirmedShardLenMax&target=xxx.cache.maxSize&now=1538085628"} 2018-09-27T17:00:28.996-0500 DEBUG render querying servers {"memory_usage_bytes": 0, "handler": "render", "carbonzipper_uuid": "7514f628-e9ec-41f3-b287-dcadf244e7db", "carbonapi_uuid": "", "handler": "multiGet", "servers": ["http://172.16.0.162:8080"], "uri": "/render/?format=protobuf&from=1538085328&target=xxx.cache.notConfirmedShardLenMax&until=1538085628"} 2018-09-27T17:00:28.997-0500 DEBUG zipper_render decoded response {"name": "xxx.cache.notConfirmedShardLenMax", "decoded": [{"name":"xxx.cache.notConfirmedShardLenMax","startTime":1538085600,"stopTime":1538085900,"stepTime":300,"values":[4],"isAbsent":[false]}]} 2018-09-27T17:00:28.997-0500 DEBUG zipper_render only one decoded response to merge {"name": "xxx.cache.notConfirmedShardLenMax"} 2018-09-27T17:00:28.997-0500 INFO access request served {"handler": "render", "carbonzipper_uuid": "7514f628-e9ec-41f3-b287-dcadf244e7db", "carbonapi_uuid": "", "format": "pickle", "target": "xxx.cache.notConfirmedShardLenMax", "memory_usage_bytes": 0, "http_code": 200, "runtime_seconds": 0.001299449}

The same version provides results when queried from graphite 0.9.15 but not 1.1.3 . Is there anything that has to be enabled on zipper side for compatibility?

deniszh commented 5 years ago

The same version provides results when queried from graphite 0.9.15 but not 1.1.3

Yes, because 1.1.3 utilize multiple target parameters in URL which zipper 0.7.x not supporting.

Is there anything that has to be enabled on zipper side for compatibility?

Try the latest zipper, this will probably work.

Civil commented 5 years ago

I think I can backport that to the 0.74 (or rather implement it there from scratch), as latest master is still not considered to be fully stable (though worth to try, it might work for you just ok)

dreddynarahari commented 5 years ago

@Civil @deniszh What are your suggestions for graphite-web + carbonzipper version compatibility? Also @Civil when will 0.74 with 1.1.3 support be available?

deniszh commented 5 years ago

@dreddynarahari Only carbonzipper 1.0.x-alpha should be compatible with the latest graphite-web. You can try it now - if it works you don't need to wait until @Civil find time to backport this feature to 0.7x

Civil commented 5 years ago

I've pushed changes to a separate branch: v0.x

But I won't build packages with this commit, at least not yet. So feel free to pick up version from there and build it yourself.