search

gnocchixyz / gnocchi

Timeseries database
Apache License 2.0
302 stars 85 forks source link

/healthcheck does not report problem #958

Open czunker opened 6 years ago

czunker commented 6 years ago

I'm not sure whether this is a gnocchi problem or oslo_middleware. We are using Gnocchi in an OpenStack setup (openstack-ansible with Pike) with redis as incoming storage. Recently we had the problem, that redis died and gnocchi didn't answer API requests. a request to metric status was answered by a HTTP 500. Our HAProxy in front of the gnocchi API is configured to query /healthcheck. But it didn't report the backends as down.

Which version of Gnocchi are you using

gnocchi-4.0.5.dev8

How to reproduce your problem

Kill redis server, used as incoming storage. But that's just an example

What is the result that you get

/healthcheck returns HTTP 200

What is result that you expected

According to the oslo docs, I would expect a HTTP 503

sileht commented 6 years ago

Gnocchi does not implements any advanced feature of oslo.middleware.healthcheck.

So by default it always return 200.

The only available oslo.middleware.healthcheck backends are disable_by_file, disable_by_files_ports

czunker commented 6 years ago

Thanks for the info. We will then work around that in another way.

jd commented 6 years ago

We should replace that middleware by something better that actually implements what @czunker expects.