Users receive macOS profile but fleetd agent is never installed

[nonpunctual]

Place holder until customer meeting

Fleet version: 4.49.0 <!-- Copy this from the "My account" page in the Fleet UI, or run fleetctl --version -->

Web browser and operating system:

across all macOS versions

---

💥  Actual behavior

Problem : When the users receive and install the macOS profile, fleet agent is never installed

Context : All our Fleet instances were updated to v4.49.0 on April 25th, it seems the bug appeared with this release. We updated all of our Fleet instance to v4.49.3 yesterday and nothing changed, bug is still here.

Impact : 16 different customers impacted. For each company, a good portion of their macOS are in this weird state of partial enrollment.

It seems that this issue happens in almost 100% of the cases for new customers.

a new customer of customer-preston is doing a POC with Fleet & all of their macOS devices are in this state. It's almost impossible to justify because enrollment is the first thing that they do.

🧑‍💻  Steps to reproduce

2. TODO

🕯️ More info (optional)

Customer has shared .mobileconfig file & logs in customer channel

Additional info / Hypotheses : We went through Fleet’s code, and it seems there were changes to the MDM lifecycle 3 weeks ago: https://github.com/fleetdm/fleet/commit/05719633a182c65fe843a05f42aaa8c648b53ee9 @roperzh See #18510

[roperzh]

Issue description

1. Fleet sends the InstallEnterpriseApplication command to install fleetd
2. The command is acknowledged by the host, but later on when it actually does the install, I can see this in the host's mdmclient logs:

 Error Domain=ASDErrorDomain Code=710 "Invalid hash 'a0a9f98ff370e632e342d9b4bfbe4f49a003a293f748891383a6cdf9ee1a53e6' expected '9fd70251a82b1b1c424d044715f69b45cb3520627ea69707f2bb2b88f78f71f2'" UserInfo={NSDebugDescription=Invalid hash 'a0a9f98ff370e632e342d9b4bfbe4f49a003a293f748891383a6cdf9ee1a53e6' expected '9fd70251a82b1b1c424d044715f69b45cb3520627ea69707f2bb2b88f78f71f2'}

I think the root problem is that when we updated fleetd as part of https://github.com/fleetdm/fleet/issues/16347, we didn't update https://download.fleetdm.com/fleetd-base-manifest.plist to have the new checksum.

Scope

This affects the flow described in the issue, and also any automatic enrollments

Remediation

1. Fleet needs to update https://download.fleetdm.com/fleetd-base-manifest.plist with the right hash
2. Customers need to send an MDM command to install fleetd to the affected hosts via the API or fleetctl
3. The command to run is:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Command</key>
    <dict>
      <key>ManifestURL</key>
      <string>https://download.fleetdm.com/fleetd-base-manifest.plist</string>
      <key>RequestType</key>
      <string>InstallEnterpriseApplication</string>
    </dict>

    <key>CommandUUID</key>
    <string>9c8195fc-218c-433e-b6f0-38e53d8c068f</string>
  </dict>
</plist>

[roperzh]

Update:

• https://download.fleetdm.com/fleetd-base-manifest.plist has been updated with the correct hash
• I have tested the workaround described above and it works

[nonpunctual]

If there is no more feedback from customer-preston by EOD 20240517 we will close. Keeping open for the rest of today in case they report further problems. Thanks @lukeheath @roperzh @sharon-fdm @getvictor @georgekarrv & the team for your fast action.

[nonpunctual]

@lukeheath Customer-preston reported that the fix for this issue is working well. No further problems reported. Closing.

[fleet-release]

Partial enrollment pain, Fleet's fix will bring full gain, Cloud city in rain.