Additional "healthz" data

pacamaster commented 8 months ago

How might this have a positive effect on your organization? It would be beneficial and have a convenient way to know the health of the endpoints.
What is the current situation? Why does the current situation hurt? Able to see if Redis, MySQL, and Fleet are healthy. It does not hurt for the moment as there are ways to get the data through other means
What are you doing right now to work around this issue? What's non-ideal about it? Need to set up Prometheus or be already inside the containers or leverage API to get data. It Isn't ideal to configure and set up other flows to get at this additional data (Tines/lambda etc).

Problem

Goal

More info

Currently, there is not much information exposed for "healthz" endpoint. some other useful information

https://github.com/fleetdm/fleet/issues/14392 - scheduled jobs status
https://github.com/fleetdm/fleet/issues/8497 - redis debug
number of teams
number of hosts
number of hosts per team
number of hosts per windows/linux/online/offline/etc
counters around queries - number of queries, number of hosts queried, time-oriented stats around queries
counters around registration - basically, ongoing registration activity (this actually might be given by existing metrics)
Fleet version ( /version for cloud hosted clients functions)

Some of this info can be obtained from queries, but having an endpoint or fast way would be useful.

Potential solutions

JoStableford commented 8 months ago

Related to a Slack conversation

noahtalerman commented 8 months ago

there is not much information exposed for "healthz" endpoint. some other useful information

Hey @pacamaster, does the customer want to consume this data? Or, do we (Fleet organization) want to consume this data?

If it's Fleet, we have usage statistics that we can see for all paying customers. If it's the customer, maybe we can send them the usage stats?

noahtalerman commented 8 months ago

Hey @pacamaster, heads up, we didn't take this one on in the current design sprint (4.48).

Please follow up to my question in the above comment before bringing this back to feature fest. Thanks!

Patagonia121 commented 7 months ago

@noahtalerman sorry I didn't get tagged here, but the customer would like to consume this data

noahtalerman commented 7 months ago

Thanks @Patagonia121. Added the ~feature fest label.

@lukeheath I'm thinking a quick win here would be to give user a way to specify a webhook URL for the existing usage stats ping.

Optional field. If specified, Fleet sends usage stats to that URL and fleetdm.com.

What do you think the level of effort would be for something like that?

lukeheath commented 7 months ago

@noahtalerman, using the existing usage statistics feed is a good idea. Adding an optional URL to send the feed to would be low effort.

The /healthz endpoint returns 200 if MySQL and Redis are healthy. Returning additional information like job run status would be really helpful and also low effort.

noahtalerman commented 7 months ago

@pacamaster we discussed this during the last feature fest.

We decided not to work on drafts for this in the upcoming sprint (4.49)

Removing from feature fest.

fleetdm / fleet