mna
commented
1 year ago Slack retranscription (from me):
Following a comment that Tomás made here (https://github.com/fleetdm/fleet/pull/8730#discussion_r1028317296), and a growing number of validations made at Fleet startup (on fleet serve, so on each fleet instance, e.g. https://github.com/fleetdm/fleet/issues/8725#issuecomment-1332294940), I wanted to raise a larger discussion about this.
When running fleet manually at the command line (interactively, user in front of the screen), those validations make sense and are decently user-friendly: if you provide e.g. an Apple Business Manager token that is expired, you get an immediate error message and Fleet is prevented from running. Same for e.g. an expired/revoked Apple APNs certificate. It's all good, you change the flag's value to point to a valid certificate/token/etc. and start it again.
However, in automated deployments similar to the ones we do in AWS for load testing/dogfood, it is not so obvious that this is the best experience - your fleet instances will fail to start and will likely enter a loop where AWS tries to get N instances up, and they all fail repeatedly due to e.g. an expired Apple BM token in its startup configuration. Not to mention the person in charge of the deployment might not be the person that can fix the thing that is failing.
Even worse (and to the initial point that Tomas was raising), as we keep adding validations at startup involving third-party network requests (e.g. pinging an Apple API to ensure the provided certificate is valid), we may encounter network slowdowns that make the check take too much time, resulting in AWS thinking the Fleet instance didn't come up, killing the instance and trying to spin up another one. Similarly, it could fail to start due to a transient network failure, even though the certificate/token/etc. that we wanted to validate is valid.
And even if there were no issues at startup, that validation is only partially useful - minutes later, the token that was valid at startup could be expired, or invalidated, the certificate could be revoked, etc. We have no good, unified way to capture those "runtime" issues and bring them up to the user's attention (basically, I think it just ends up in the logs). We have a ticket planned to show a banner when Apple BM terms have changed and the user needs to accept them (similar to how we show a banner if the fleet license is expired), but this is special-casing one of the many potential failures (https://github.com/fleetdm/fleet/issues/8537).
I wonder if we should plan a Notification system where all such issues can bubble up and be brought to the user's attention. I can think of many failures that could benefit from that, in addition to the Apple's terms changing:
- Apple BM server token expiring
- Apple BM server token being invalidated due to a change of password of the token's account
- Apple APNs/SCEP certificate expiring/being revoked
- MDM enrollment's default team not existing (we validate that the team exists when setting the default team, but nothing prevents the team from being deleted/renamed, and we match it by name), resulting in failure to add newly enrolled hosts in a team
That's just thinking about recent MDM-related stuff, I'm sure there are failure modes in the cron jobs that would be worth bringing forth in a notification system.
With that in place, I think we could limit the startup validations to just the required dependencies of Fleet itself (i.e. mysql, redis, ...) and anything third-party could be lazily validated on first use and raised as a notification, avoiding any network-related issues at startup (whether its the slowdown, the transient errors, etc.). And I guess it would be easier to bring the issue to the right person that way (e.g. the person that can renew the Apple BM token).
From @chiiph :
it sounds like we'll want to have a "health check" cron that can populate a table with last checks that we can display in fleetctl. We can check as many things as we want here
that will also help a lot with debugging customer's deployments
as we add more options and moving parts, we'll need to improve introspection in a way like this
unless anybody is drastically opposed to this, this sounds like something that should go on a ticket, and we can work on it as part of the overall "holistic underlying tech stuff for mdm"
noahtalerman
roperzh
nonpunctual
fleet-release
Problem
As we add more interactions with third-party systems (e.g. Apple MDM) and more asynchronous work (cron jobs, MDM commands, etc.), the need for a way to surface important failures (or even more generally results) to the users is growing and we don't have a unified way to do this currently.
For example, we may validate at startup that the provided Apple Business Manager token is valid, but it may become invalid soon after, or Apple's terms may have changed and need to be accepted by the user before the token starts working again, etc. Many such failure scenarios require a user intervention and possibly a configuration change, and may happen at any time after the fleet instances startup.
Requirements
TBD. This is a reference ticket to keep track of this feature.
Potential Solutions
See comments below for a transcription of the slack thread for reference.