Enforce OS Profiles Stuck on enforcing - macOS

[nonpunctual]

Fleet version: <!-- Copy this from the "My account" page in the Fleet UI, or run fleetctl --version -->

{
  "version": "4.58.0",
  "branch": "HEAD",
  "revision": "e98f86d0c87979a48e0816e5c55be3602de5b7e5",
  "go_version": "go1.23.1",
  "build_date": "2024-10-17",
  "build_user": "runner"
}

Web browser and operating system:

NA

💥  Actual behavior

The following profiles are stuck in "Enforcing" state.

The fact that a custom os config file stays in Enforcing (pending) while device being online, seems to trigger a huge load on the RDS Database instance

Auto Install macOS Updates Nov 14 2024.mobileconfig

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>PayloadContent</key>
    <array>
        <dict>
            <key>AllowPreReleaseInstallation</key>
            <true/>
            <key>AutomaticCheckEnabled</key>
            <true/>
            <key>AutomaticDownload</key>
            <true/>
            <key>AutomaticallyInstallAppUpdates</key>
            <true/>
            <key>AutomaticallyInstallMacOSUpdates</key>
            <true/>
            <key>ConfigDataInstall</key>
            <true/>
            <key>CriticalUpdateInstall</key>
            <true/>
            <key>SUDisableEVCheck</key>
            <false/>
            <key>forceDelayedSoftwareUpdates</key>
            <false/>
            <key>restrict-software-update-require-admin-to-install</key>
            <false/>
            <key>PayloadDisplayName</key>
            <string>Software Update</string>
            <key>PayloadIdentifier</key>
            <string>com.apple.SoftwareUpdate.08F9E289-0308-4E68-B9F5-44C8C621054B</string>
            <key>PayloadType</key>
            <string>com.apple.SoftwareUpdate</string>
            <key>PayloadUUID</key>
            <string>08F9E289-0308-4E68-B9F5-44C8C621054B</string>
            <key>PayloadVersion</key>
            <integer>1</integer>
        </dict>
    </array>
    <key>PayloadDisplayName</key>
    <string>AUTO_INSTALL_MACOS_UPDATES</string>
    <key>PayloadIdentifier</key>
    <string>com.primo.softwareupdate</string>
    <key>PayloadOrganization</key>
    <string>Primo</string>
    <key>PayloadRemovalDisallowed</key>
    <true/>
    <key>PayloadScope</key>
    <string>System</string>
    <key>PayloadType</key>
    <string>Configuration</string>
    <key>PayloadUUID</key>
    <string>4095D18C-9711-4B4D-BE31-3C24045C8041</string>
    <key>PayloadVersion</key>
    <integer>1</integer>
</dict>
</plist>

Auto Install macOS Updates.mobileconfig

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>PayloadContent</key>
    <array>
        <dict>
            <key>AllowPreReleaseInstallation</key>
            <false/>
            <key>AutomaticCheckEnabled</key>
            <true/>
            <key>AutomaticDownload</key>
            <true/>
            <key>AutomaticallyInstallAppUpdates</key>
            <true/>
            <key>AutomaticallyInstallMacOSUpdates</key>
            <false/>
            <key>ConfigDataInstall</key>
            <true/>
            <key>CriticalUpdateInstall</key>
            <true/>
            <key>PayloadDisplayName</key>
            <string>Software Update</string>
            <key>PayloadIdentifier</key>
            <string>com.apple.SoftwareUpdate.94017872-E43C-474F-BDFB-B92CA2DA2D2E</string>
            <key>PayloadType</key>
            <string>com.apple.SoftwareUpdate</string>
            <key>PayloadUUID</key>
            <string>41D36EDA-D723-4D7D-B499-FCC4E3136D49</string>
            <key>PayloadVersion</key>
            <integer>1</integer>
            <key>SUDisableEVCheck</key>
            <false/>
            <key>forceDelayedSoftwareUpdates</key>
            <false/>
            <key>restrict-software-update-require-admin-to-install</key>
            <false/>
        </dict>
    </array>
    <key>PayloadDisplayName</key>
    <string>AUTO_INSTALL_MACOS_UPDATES_APPS_AND_SECURITY</string>
    <key>PayloadIdentifier</key>
    <string>com.primo.softwareupdate.apps.security</string>
    <key>PayloadOrganization</key>
    <string>Primo</string>
    <key>PayloadRemovalDisallowed</key>
    <true/>
    <key>PayloadScope</key>
    <string>System</string>
    <key>PayloadType</key>
    <string>Configuration</string>
    <key>PayloadUUID</key>
    <string>922F3373-D4B7-4125-ACCB-0E9A43A13B06</string>
    <key>PayloadVersion</key>
    <integer>1</integer>
</dict>
</plist>

🧑‍💻  Steps to reproduce

1. We upgraded the Mysql instance to be double the initial size (per recommendation from Fleet CS & infra engineering)
2. This did fix the Mysql cluster lags
3. BUT the root problem is still here (see above)

What’s the impact :

• Customer "HomeExchange" still have 100% of their Mac with the Enforcing (pending) state
• Big network issue were reported by this customer after our call this afternoon
• 50+ of their employee are experiencing disconnection on their network and losing access to the internet

Current resolution :

• We removed the OS Config profile that was stuck in Enforcing (pending)
• It imediatly resolved the network issues for the customer
• Adding the profile on device ONE by ONE seems to work (using scoping by label)
• Adding the profile on all the devices at once make them all stuck and they all stay in Enforcing (pending)

Conclusion based on our experience:

• Profile in Enforcing (pending) are causing huge network load
• The company’s employee are probably all connected to the same network
• Upgrading the database cluster fixed the overall lags but root problem is still impacting us

[nonpunctual]

@lukeheath @zayhanlon Needs prioritization. This is impacting network & computer performance at MSP customer.

[zayhanlon]

@lukeheath are you okay with a p2 here?

[lukeheath]

@zayhanlon Yep, this is a P2. Adding to the release board.

[nonpunctual]

cc @jahzielv @gillespi314

[nonpunctual]

Questions:

• Are these customized OS update profiles incompatible with Fleet's macOS Update features?
• Are network issues at the MSP customer due to large numbers of simultaneous downloads from Apple for OS updates?
• Are there iOS devices enrolled in this environment?
• Is APNS getting rate-limited because of too many requests?
• Excessive "not now" / offline from Hosts? May mean all Hosts checked in at once...