Add a new /health endpoint to validate all components are ready

[lucasteligioridis]

We'd love a /health endpoint that validates all the services for Faktory are working, which would include and not limited to:

• Back end
• Web UI
• Redis connectivity
• License validity (if running Ent or Pro)
• Server version

This would mean we could use this endpoint to check the complete health of the faktory-server and everything is required to be "working". With the license availability, we'd still expect that to check once a day and just change then, to prevent an overwhelm of external calls and to your DL server, we might be hitting this endpoint once every 5 seconds.

This could then be updated in the documentation, I'd prefer to make http calls for health than the TCP socket, since this would be checking the same thing anyway :)

[mperham]

There are two endpoints that might be useful:

• The /stats JSON endpoint in the Web UI:

  ❯ curl http://localhost:7420/stats
  {"faktory":{"queues":{"bulk":3,"critical":4,"default":122,"fwgtest":1},"tasks":{"Busy":{"reaped":0,"size":0},"Dead":{"cycles":0,"enqueued":0,"size":0,"wall_time_sec":0},"Retries":{"cycles":7,"enqueued":0,"size":0,"wall_time_sec":0.001150302},"Scheduled":{"cycles":7,"enqueued":0,"size":0,"wall_time_sec":0.005341324},"Workers":{"reaped":0,"size":0}},"total_enqueued":130,"total_failures":503,"total_processed":1704,"total_queues":4},"server":{"command_count":0,"connections":0,"description":"Faktory","faktory_version":"1.5.1","uptime":31,"used_memory_mb":"68 MB"},"server_utc_time":"14:34:46 UTC"}

• The INFO command can be used by any Faktory client to retrieve the same data.

  ❯ ./faktory-cli
  Connected to Faktory 1.5.1
  > INFO
  {"faktory":{"queues":{"bulk":3,"critical":4,"default":122,"fwgtest":1},"tasks":{"Busy":{"reaped":0,"size":0},"Dead":{"cycles":5,"enqueued":0,"size":0,"wall_time_sec":0.000285788},"Retries":{"cycles":59,"enqueued":0,"size":0,"wall_time_sec":0.007235047},"Scheduled":{"cycles":59,"enqueued":0,"size":0,"wall_time_sec":0.016630973},"Workers":{"reaped":0,"size":0}},"total_enqueued":130,"total_failures":503,"total_processed":1704,"total_queues":4},"server":{"command_count":7,"connections":1,"description":"Faktory","faktory_version":"1.5.1","uptime":292,"used_memory_mb":"68 MB"},"server_utc_time":"14:39:07 UTC"}

They don't give any data about the license because they are both meant to be used by any Faktory instance.

[lucasteligioridis]

Happy to use the /stats endpoint, which I'm currently using as a health check.

Can you clarify with me that if the /stats endpoint returns at all, it is a good indication that everything has connected correctly? i.e. redis / faktory backend are up?

Any chance we could get some metadata about redis in there?

[mperham]

What redis info would you want to see?

[lucasteligioridis]

Maybe a validation that redis has connected.

Would also include something about the redis host name? i.e. to validate that it connected to either our own one or the one baked into the faktory server.

[mperham]

The /stats endpoint pulls the queue sizes so each call does implicitly hit Redis, it will return 500 if Redis is down:

❯ curl -i http://localhost:7420/stats
HTTP/1.1 500 Internal Server Error
Content-Language: en
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Thu, 15 Apr 2021 22:05:35 GMT
Content-Length: 53

dial tcp 127.0.0.1:6379: connect: connection refused

The /debug page shows your Redis location or URL in Data Location:

[lucasteligioridis]

Ah great, so it's an implicit check of redis apart of the /stats call 👍🏼 I'm happy with that result. Can always query /debug for more information as you suggested 👍🏼