GitOps fails when there are unaccepted Apple Business Manager terms & conditions

noahtalerman commented 2 months ago

@noahtalerman: deebradel requested this because they manage 100 clients in one GitOps repo. When the ABM terms expire, GitOps errors and doesn't allow changes for all clients. Clients are responsible for accepting the new ABM terms so deebradel has to ask each client, at different time, to accept the new terms. This blocked deployments for up to 18 hours across 70 organizations, making GitOps unreliable for timely updates.
- @noahtalerman: In the interim they manually retried GitOps runs after each failed attempt, but kept encountering new T&C batches that hadn’t been accepted yet, creating operational delays and confusion.
- @noahtalerman: Eventually Fleet could change GitOps so that it tells you which ABM token needs new terms accepted.
@noahtalerman: Today, in Settings > Integrations > Automatic enrollment, Fleet will show a caution icon next to the Apple Business Manager that has expired terms:
@bettapizza: The staggered rollout could take 24-72 hours to reach all customers.
- The email from Apple goes out to all customers at once.

customer-numa: would be nice to have an error during dry run june 2025 update from @zayhanlon

@ksatter: numa ran into this error:

GET[ https://customer.fleetdm.com/api/latest/fleet/mdm/bootstrap/7/metadata?for_update=true](https://customer.fleetdm.com/api/latest/fleet/mdm/bootstrap/7/metadata?for_update=true) 404 Not Found (78ms)
{
  "message": "Validation Failed",
  "errors": [
    {
      "name": "team_id",
      "reason": "bootstrap package for this team does not exist"
    }
  ]
}
POST[ https://customer.fleetdm.com/api/latest/fleet/enrollment_profiles/automatic](https://customer.fleetdm.com/api/latest/fleet/enrollment_profiles/automatic)
{"team_id":7,"name":"../profiles/mac/_default-enrollment.dep.json","enrollment_profile":{"profile_name":"Customer default macOS enrollment profile","region":"US","language":"en","org_magic":"1","allow_pairing":true,"is_supervised":true,"is_mdm_removable":false,"skip_setup_items":["AppleID","AppStore","Diagnostics","iCloudDiagnostics","iCloudStorage","Intelligence","Location","Payment","Privacy","Restore","ScreenTime","Siri","TermsOfAddress","TOS","UnlockWithWatch","Welcome"]}}
POST[ https://customer.fleetdm.com/api/latest/fleet/enrollment_profiles/automatic](https://customer.fleetdm.com/api/latest/fleet/enrollment_profiles/automatic) 422 Unprocessable Entity (147ms)
{
  "message": "Validation Failed",
  "errors": [
    {
      "name": "profile",
      "reason": "sending profile to Apple failed: Post \"https://mdmenrollment.apple.com/profile\": DEP auth error: 403 Forbidden: T_C_NOT_SIGNED"
    }
  ],
  "uuid": "9d8bb01e-b110-44f4-8637-1acff9d26b16"
}
Error: uploading macOS setup assistant for team "Endpoints (canary)": POST /api/latest/fleet/enrollment_profiles/automatic received status 422 Validation Failed: sending profile to Apple failed: Post "https://mdmenrollment.apple.com/profile": DEP auth error: 403 Forbidden: T_C_NOT_SIGNED
Running into errors on GitOps run

noahtalerman commented 2 months ago

FYI @ddribeiro

noahtalerman commented 1 month ago

Here's another request:

70 organizations Couldn’t do GitOps deployments without accepting T&C => Dale: Apple updated their ABM terms and conditions. Hard problem for deebradel because new terms get rolled at a rolling basis meaning some customers can accept new terms before others => Dale: GitOps failed meaning no updates happened => Noah: What’s the error message?

deebradel: Is there any way to avoid or should we check this before the run every time?

Potential solutions:

Have the run fail immediately if there are T&C that need to be accepted
The total time that they weren’t able to run GitOps was 18 hours because the T&C acceptance was pushed in batches of ~5 customers. Every time they would wait a small stretch and then try again, they would hit another subset of customers that had new T&C

Kathy: numa ran into the same issue:

GET[ https://customer.fleetdm.com/api/latest/fleet/mdm/bootstrap/7/metadata?for_update=true](https://customer.fleetdm.com/api/latest/fleet/mdm/bootstrap/7/metadata?for_update=true) 404 Not Found (78ms)
{
  "message": "Validation Failed",
  "errors": [
    {
      "name": "team_id",
      "reason": "bootstrap package for this team does not exist"
    }
  ]
}
POST[ https://customer.fleetdm.com/api/latest/fleet/enrollment_profiles/automatic](https://customer.fleetdm.com/api/latest/fleet/enrollment_profiles/automatic)
{"team_id":7,"name":"../profiles/mac/_default-enrollment.dep.json","enrollment_profile":{"profile_name":"Customer default macOS enrollment profile","region":"US","language":"en","org_magic":"1","allow_pairing":true,"is_supervised":true,"is_mdm_removable":false,"skip_setup_items":["AppleID","AppStore","Diagnostics","iCloudDiagnostics","iCloudStorage","Intelligence","Location","Payment","Privacy","Restore","ScreenTime","Siri","TermsOfAddress","TOS","UnlockWithWatch","Welcome"]}}
POST[ https://customer.fleetdm.com/api/latest/fleet/enrollment_profiles/automatic](https://customer.fleetdm.com/api/latest/fleet/enrollment_profiles/automatic) 422 Unprocessable Entity (147ms)
{
  "message": "Validation Failed",
  "errors": [
    {
      "name": "profile",
      "reason": "sending profile to Apple failed: Post \"https://mdmenrollment.apple.com/profile\": DEP auth error: 403 Forbidden: T_C_NOT_SIGNED"
    }
  ],
  "uuid": "9d8bb01e-b110-44f4-8637-1acff9d26b16"
}
Error: uploading macOS setup assistant for team "Endpoints (canary)": POST /api/latest/fleet/enrollment_profiles/automatic received status 422 Validation Failed: sending profile to Apple failed: Post "https://mdmenrollment.apple.com/profile": DEP auth error: 403 Forbidden: T_C_NOT_SIGNED
Running into errors on GitOps run

fleetdm / fleet

GitOps fails when there are unaccepted Apple Business Manager terms & conditions #28350