Assigning automatic enrollment (DEP) profile sometimes fails

roperzh commented 11 months ago

Fleet version: 4.39.0

💥 Actual behavior

When a macOS host is assigned a automatic enrollment (DEP) profile using Apple's AssignProfile API, the Apple server sometimes return a 200 response w/ an error in response body:

{
  "profile_uuid": "9145EDB932A42C48A9DD56AD69213C64",
  "devices": [{"REDACTED_SERIAL_NUMBER": "FAILED"}]
}

Fleet is currently ignoring the response body, and we need to understand:

What to do when we get that error
How to prevent that error state

🧑‍💻 Steps to reproduce

Unclear what are the steps, but this is what I (Roberto) did:

Add a DEP profile to "No team"
Create a team and add a DEP profile to that team
Move a macOS host from "No team" to the team with the custom DEP profile

🕯️ More info

The below fix is what we've heard has worked in the past from a reputable developer whose built open-source MDM solutions:

I would suggest you put the serials that returned FAILED on a 24 hour cooldown and retry in 24 hours. Usually that resets any server side rate limits you might have run into.

🛠️ To fix

Now:

Hardcode the serial number for the host having issues, wait 24 hours after last FAILED response for this host before retrying to assign the profile.
Add this fix to a new branch off of ~~4.39.0~~ 4.40
Deploy this branch for the customer

Later:

For any DEP profile assignment that returns a FAILED response, wait 24 hours after the FAILED response before retrying to assign the profile.

UPDATE: As of now, no devices are failing DEP profile assignment. Customer used the wait 24 hours workaround for devices in prod. Seems like we (Fleet) need to repro ourselves. Plan is to close the bug and prioritize this story as a next step: https://github.com/fleetdm/fleet/issues/15461