Takes too long to clear ABM terms banner

[noahtalerman]

Fleet version: Observed in Fleet's dogfood environment (commit 8fb694f)

---

🧑‍💻  Expected behavior

Apple Business Manager (ABM) terms expired. I saw the new terms banner in the Fleet UI. I accepted the new terms on ABM.

I navigated to the Hosts page in Fleet and waited 60 seconds. I refreshed the page. I expected the ABM terms banner in Fleet to be gone because the default interval in which Fleet talks to ABM is 60 seconds (see configuration option here).

💥  Actual behavior

It took ~15 mins for the ABM terms banner to dissapear.

More info

A customer also experienced this delay. It took 1 hr+ for the banner to clear.

This customer also had something weird going on with their ABM connection. More in Slack here (internal). Separate issue is captured here: #12958

Code where we connect to DEP: https://github.com/fleetdm/fleet/blob/09e6bf98079cd66eeb0cd8b0f0ba5801ebbebb9e/server/mdm/apple/apple_mdm.go#L531

Original story for the ABM terms banner feature: #8537

[noahtalerman]

@gillespi314 when does Fleet hit the DEP API and clear the flag? If we hit the API on every page in the UI we might want to update the copy in the banner.

Currently the copy tells the IT admin to navigate to the Hosts page to clear the banner. For reference, I saw the banner clear when I navigated to the Host details page.

[gillespi314]

when does Fleet hit the DEP API and clear the flag?

By my reading, this happens when a server starts up and then every DEP-sync loop (60 seconds by default). If it detects a change, it is supposed to update the flag in the app config. It's not tied directly to any UI actions.

In most cases, UI reloads the app config when the user navigates between main tabs. There might be some exceptions that I can't recall right now.

[noahtalerman]

Update from Martin: IIRC the flag would clear on the next successful API call to Apple's DEP API (once the terms have been accepted). I'm not sure how often/at what frequency those calls are made, though.

[ireedy]

Bug has aged out. Moving back to product drafting.

[georgekarrv]

Timebox this to trying to find the missing link for ~2hr otherwise would love to bring a user action button through design if we can't easily find this.

[mna]

Uses of the Apple DEP API Client

• The Worker Job processor for the MacosSetupAssistant job (to assign the profile to devices, calls AssignProfile)
• The GET /api/_version_/fleet/mdm/apple_bm endpoint (calls AccountDetail)
• The GET /api/_version_/fleet/mdm/apple/dep/devices endpoint (calls FetchDevices)
• In RegisterProfileWithAppleDEPServer to ensure the setup assistant (default or custom) is registered with Apple's DEP server. Called from the Worker job mentioned above, and also from the syncer cron job. (calls DefineProfile)
• The RunAssigner cron job (job name: "apple_mdm_dep_profile_assigner"), which runs every 1m by default (the mdm.apple_dep_sync_periodicity config setting). Is guaranteed to call at least FetchDevices, may also call SyncDevices (unless there's a critical error e.g. database down). This is via nanodep's Syncer.Run method.

Caching of AppConfig

• On the server, the AppConfig is cached (read from memory in the Fleet instance instead of from the DB) for up to 1s.
• Is it cached on the frontend too? How often is it refreshed?
  • Given that the ticket's description mentions: "I navigated to the Hosts page in Fleet and waited 60 seconds. I refreshed the page.", it's unlikely that a client cache is causing the issue?

Logging of Terms-related Actions

• When running with "--debug" logging enabled, the Fleet server will log Apple DEP client: updated app config Terms Expired flag whenever the flag state changes (along with the new value true or false of the flag).
• If the terms are still not accepted when the syncing cron job runs, it will log in Info level the message "error syncing" with the actual error (which should be a 403 status code with a message containing T_C_NOT_SIGNED).

Potential Causes

The fact that the banner does show up and disappear eventually would indicate that the logic and behavior around detecting the terms changes is working as intended.

• The customer's 1h delay seems to have been caused by other issues - they do mention that the ABM sync did not happen for over an hour, which would align with some issue contacting ABM in our sync cron job, which would explain why the banner could not be cleared (it needs a successful DEP API request to clear).
• They also mention the banner disappearing properly (faster) in a different environment.
• In the case Noah mentions in this ticket, it took around 15 minutes.
• We have a cron job that runs (by default) every minute and guarantees at least 1 request to Apple's DEP API, which results in a check and update of the terms flag if those have been detected as accepted. Other places where we call Apple's DEP API are not guaranteed to run regularly.

In light of this:

• It is possible that it takes a while for the terms agreement to be reflected in Apple's API requests. This is not something we can control.
• We should check the logs the next time the terms change and require an agreement, and get a clear picture of the time-frame of the related actions - I'll push a PR to change the logging of when the terms flag is updated in Fleet so that it does not require debug logging (it's not something that happens often anyway so it won't spam the logs).
• There's still a small possibility that the frontend caches the appconfig (if the page refresh mentioned in the ticket's description is not a "full browser refresh", but just e.g. navigating to a different page and coming back.
  • Someone familiar with the frontend should take a look at this and confirm if the appconfig gets refreshed often enough for this scenario.

[mna]

@noahtalerman @georgekarrv See my comment above for my findings in the timeboxed investigation.

tl;dr; a) I created and merged a PR which logs without requiring Debug logging when the terms flag changes, b) someone from the frontend should confirm that there's no caching issue on the frontend, c) next terms change we should grab the logs and look at the time-frame of events.

It's possible that ~15 minutes is the time it takes for Apple to reflect the newly-agreed state in the API requests.

Should I move this to "ready for release" as there's nothing to QA for now, or should we keep the ticket active so we do the follow-up steps next time the terms change?

[mna]

We were able to test this through a real ABM terms change almost immediately, and we could see that:

1. The cron job calls the Apple API each minute as expected;
2. Once the terms were accepted in ABM, Apple kept on replying to the API calls with the T_C_NOT_SIGNED error for over 25 minutes;
3. Once Apple's API stop returning that error, the banner promptly disappeared from the Web page, so there's no frontend caching that is causing other delays

So the reason for the delay is out of our control, due to how Apple keeps sending the T_C_NOT_SIGNED error long after the terms were accepted. We should handle that through UX/UI, I'll assign the ticket to @noahtalerman to review the banner text.

[noahtalerman]

Thanks Martin!

Hey @marko-lisica passing this one to you. Do you think we should update the copy?

[marko-lisica]

Private Zenhub Image

The decision for this one is to change copy only to make clear that user might need to wait some time for banner to disappear. (see image above)

New copy: Your organization can’t automatically enroll macOS hosts until you accept the new terms and conditions for Apple Business Manager (ABM). An ABM administrator can accept these terms. Done? It might take some time for ABM to report back to Fleet.

@mna @noahtalerman @georgekarrv

[sabrinabuckets]

Due to the nature of this issue—only being testable when Apple releases new T&Cs—it isn't immediately testable. I have reviewed the copy changes in the linked PR and am comfortable with the decisions made.

[noahtalerman]

@sabrinabuckets FWIW I think you can test this by manually setting a specific flag.

[fleet-release]

Banner lingers on, Swift sync brings clarity, Fleet's time now aligns.