Traditional LaunchAgent doesn't load with 1.1.12.81501

[kevinmcox]

After updating to Nudge 1.1.12.81501 on macOS 14 Sonoma the traditional LaunchAgent no longer loads on boot/login.

/bin/launchctl list | grep Nudge no longer shows the LaunchAgent as loaded.

/bin/launchctl load /Library/LaunchAgents/com.github.macadmins.Nudge.plist returns success, but does not actually succeed in loading the LaunchAgent.

Installing the Nudge_LaunchAgent-1.0.1.pkg package does not result in the LaunchAgent being loaded.

Double-clicking Nudge.app however does load the LaunchAgent. After that launchctl list shows it as loaded.

However after a reboot it is unloaded once again.

I have tested this scenario with a configuration profile requiring 14.99 and Nudge never launches on it's own. Opening Nudge manually results in the expected behavior, but if you select a deferral option or reboot the LaunchAgent is no longer loaded.

The Nudge Logger LaunchDaemon works as expected.

[sundeep443]

We were also facing the same issue. Is there any options to rectify this as our monthly os patching is getting affected.

[erikng]

I didn't change the code signing or launch agent so I'll need more information. Does downgrading nudge fix the issue?

Can someone try and gather logs from launchctl and see why it's failing to load?

[kevinmcox]

I'm assuming this is related to the SMAppService addition.

Now that I know it isn't an isolated issue I'll do additional testing tomorrow.

[sundeep443]

I have downgraded but still automatically nudge not displaying. Will check further and update.

[erikng]

Kevin, that code does nothing unless it's invoked. It's off by default and an opt-in only setting.

[erikng]

I have downgraded but still automatically nudge not displaying. Will check further and update.

Thanks for confirming. I am highly suspect of the issue being the new nudge version and is something else.

There have been reports of nudge not launching if you upgrade from a version with the old cert to the new cert. I wonder if that's what is happening here.

[kevinmcox]

(There is some splintered discussion on this in Slack.)

In my case I'm coming from the previous release which was already signed with the new certificate.

[timnottom]

For what it's worth: I've also tried starting the launchagent by doing a launchctl bootstrap gui/502 /Library/LaunchAgents/com.github.macadmins.Nudge.plist and/or sudo launchctl bootstrap gui/502 /Library/LaunchAgents/com.github.macadmins.Nudge.plist

This didn't make a difference.

[tuxudo]

I'm seeing this on my fleet as well. They've only ever run Nudge 1.1.12. We were able to Nudge from 13.x to 13.5.2 without an issue. But Nudge is not appearing for Macs running 13.5.2 to nudge them to 13.6/14.0.

Digging in the log at /var/log/com.apple.xpc.launchd/launchd.log, I see launchd complaining about the Nudge LA:

2023-10-05 11:03:31.115265 (gui/502/com.github.macadmins.Nudge) <Notice>: internal event: WILL_SPAWN, code = 0
2023-10-05 11:03:31.115287 (gui/502/com.github.macadmins.Nudge) <Notice>: service state: spawn scheduled
2023-10-05 11:03:31.115288 (gui/502/com.github.macadmins.Nudge) <Notice>: service state: spawning
2023-10-05 11:03:31.115333 (gui/502/com.github.macadmins.Nudge) <Notice>: launching: speculative
2023-10-05 11:03:31.115967 (gui/502/com.github.macadmins.Nudge [14037]) <Notice>: xpcproxy spawned with pid 14037
2023-10-05 11:03:31.115982 (gui/502/com.github.macadmins.Nudge [14037]) <Notice>: internal event: SPAWNED, code = 0
2023-10-05 11:03:31.115985 (gui/502/com.github.macadmins.Nudge [14037]) <Notice>: service state: xpcproxy
2023-10-05 11:03:31.116081 (gui/502 [100040]) <Notice>: Bootstrap by launchctl[14036] for /Library/LaunchAgents/com.github.macadmins.Nudge.plist succeeded (0: )
2023-10-05 11:03:31.116139 (gui/502/com.github.macadmins.Nudge [14037]) <Notice>: internal event: SOURCE_ATTACH, code = 0
2023-10-05 11:03:31.121639 (gui/502/com.github.macadmins.Nudge [14037]) <Notice>: Requesting first run LWCR update
2023-10-05 11:03:31.131457 (gui/502/com.github.macadmins.Nudge [14037]) <Error>: Service could not initialize: Unable to verify trusted spawn(/Applications/Utilities/Nudge.app/Contents/MacOS/Nudge, /Library/LaunchAgents/com.github.macadmins.Nudge.plist, com.github.macadmins.Nudge, 3, 502), error 0xa1 - Service cannot be launched because of BTM policy
2023-10-05 11:03:31.131478 (gui/502/com.github.macadmins.Nudge [14037]) <Error>: initialization failure: 22G120: xpcproxy + 16320 [460][5756EC64-7B45-3833-AC0B-8686748C8577]: 0xa1
2023-10-05 11:03:31.131482 (gui/502/com.github.macadmins.Nudge [14037]) <Error>: Untrusted service was denied launch by BTM. Removing.
2023-10-05 11:03:31.131486 (gui/502/com.github.macadmins.Nudge [14037]) <Notice>: internal event: INIT, code = 161
2023-10-05 11:03:31.132160 (gui/502/com.github.macadmins.Nudge [14037]) <Notice>: xpcproxy exited due to exit(78)
2023-10-05 11:03:31.132170 (gui/502/com.github.macadmins.Nudge [14037]) <Notice>: exited due to exit(78)
2023-10-05 11:03:31.132174 (gui/502/com.github.macadmins.Nudge [14037]) <Notice>: already handled failed init, ignoring
2023-10-05 11:03:31.132178 (gui/502/com.github.macadmins.Nudge [14037]) <Notice>: service state: exited
2023-10-05 11:03:31.132194 (gui/502/com.github.macadmins.Nudge [14037]) <Notice>: internal event: EXITED, code = 0
2023-10-05 11:03:31.132198 (gui/502 [100040]) <Notice>: service inactive: com.github.macadmins.Nudge
2023-10-05 11:03:31.132202 (gui/502 [100040]) <Notice>: removing service: com.github.macadmins.Nudge
2023-10-05 11:03:31.132243 (gui/502/com.github.macadmins.Nudge [14037]) <Notice>: internal event: PETRIFIED, code = 0
2023-10-05 11:03:31.132247 (gui/502/com.github.macadmins.Nudge [14037]) <Notice>: service state: not running

The error lines appear to be:

2023-10-05 11:03:31.131478 (gui/502/com.github.macadmins.Nudge [14037]) <Error>: initialization failure: 22G120: xpcproxy + 16320 [460][5756EC64-7B45-3833-AC0B-8686748C8577]: 0xa1
2023-10-05 11:03:31.131482 (gui/502/com.github.macadmins.Nudge [14037]) <Error>: Untrusted service was denied launch by BTM. Removing.

[erikng]

https://developer.apple.com/forums/thread/717609

https://support.apple.com/guide/deployment/manage-login-items-background-tasks-mac-depdca572563/web

[flammable]

We're using the default LaunchAgent:

https://github.com/macadmins/nudge/blob/main/build_assets/com.github.macadmins.Nudge.plist

That one includes AssociatedBundleIdentifiers. What else is missing?

[duderin0]

Just to add data to this, I'm starting to use Nudge for the first time.

Nudge 1.1.12.81501, test machine is MacOS 13.2.1 and keys set to 13.6 and Oct 6 at 5:00pm via Jamf Config Profile. Manually running Nudge shows correct items. I installed NudgeSuite via an Installomator script, and like I mentioned its the first time this device has seen Nudge. The launchctl list is also empty for me.

[kevinmcox]

I tested on clean, unmanaged, installs with the only config profile being Nudge.

It works as expected on 13.6 but fails on 14.0.

My logs match @tuxudo's above.

[kevinmcox]

If I edit the traditional LaunchAgent to change the Label (e.g. com.github.macadmins.Nudge2) it works as expected under Sonoma.

[kevinmcox]

The above makes me think this is a conflict with the SMAppService bundled LaunchAgent and the traditional one having the same Label.

The solution might just be as easy as changing the label on the bundled one. But admittedly I don't know how things will interact if a user has the traditional LaunchAgent installed and also tries to utilize the bundled one with different Labels.

[tuxudo]

I too can get it working if I change the LA file name and the Label key to match that of @kevinmcox. It also creates a Nudge entry in Sys Prefs' Login Items section instead of just having the Mac Admins Open Source entry

Also tested on a downgraded macOS 13.3 with 1.1.12, same issue with the normal Nudge LA.

[kevinmcox]

I didn't even have to change the filename, just edited the Label value in the plist.

[erikng]

This seems like a bug in macOS.

[kevinmcox]

Running launchctl load /Applications/Utilities/Nudge.app/Contents/Resources/com.github.macadmins.Nudge.plist works to load the bundled LA, even with the traditional one in /Library/LaunchAgents/.

So it seems like SMAppService is giving a higher priority to the bundled LA, over the traditional one, when they share a label.

[flammable]

I haven't tried it, but could we replace the traditional LaunchAgent with a symlink to /Applications/Utilities/Nudge.app/Contents/Resources/com.github.macadmins.Nudge.plist?

[kevinmcox]

Other tests around SMAppService:

Simply enabling the new Nudge loadLaunchAgent preference key doesn't actually load the bundled LaunchAgent. Neither when Nudge is installed or after a reboot. Nudge needs to be opened at least once manually to load the bundled LaunchAgent.

Interestingly, if Nudge's loadLaunchAgent preference key is set to true, then installing the traditional LaunchAgent package does succeed in loading the LaunchAgent.

[erikng]

I need some Apple people to look at this. :/ the documentation doesn't account for doing what we've done.

[kevinmcox]

@erikng check your MacAdmins Slack DMs when you get a chance.

[kevinmcox]

There might be some useful info here as well from @bartreardon: https://macadmins.slack.com/archives/C0HLW2QAH/p1686245265816419?thread_ts=1686200709.894959&cid=C0HLW2QAH

[bartreardon]

Unless Apple have made changes since June (I haven't checked) SMAppService documentation overstates how useful it is for system utilities unfortunately.

I really need to follow up with them about it now Sonoma is out and talk to someone that actually works on the thing.

[khronokernel]

To add an extra data point (don't mean to flood), can replicate on a number of machines with a fresh install in Ventura:

<Error>: Service could not initialize: Unable to verify trusted spawn(/Applications/Utilities/Nudge.app/Contents/MacOS/Nudge, /Library/LaunchAgents/com.github.macadmins.Nudge.plist, com.github.macadmins.Nudge, 3, 501), error 0xa1 - Service cannot be launched because of BTM policy
<Error>: initialization failure: 22G91: xpcproxy + 20182 [460][E5C56CFD-9EFD-3187-8D09-B350D1F17D04]: 0xa1
2023-10-05 14:05:28.890996 (gui/501/com.github.macadmins.Nudge [652]) 
<Error>: Untrusted service was denied launch by BTM. Removing.

Following models were tested:

• iMac18,2 running 13.1, configured to target 13.6
• iMac18,3 running 13.5.2, configured to target 13.6
• Apple Silicon Virtual Machine running 13.6, configured to target 14.0

And can confirm that tuxudo's Label edit does fix the issue.

[kevinmcox]

@bartreardon thank your for adding your thoughts here.

Would you agree that it seems like Apple’s intent here is to make it easy for apps to launch at login without having to ship a LaunchAgent? e.g. Things like VPN apps, menu bar utilities, etc. Apple is expecting users to manually download the software, launch once and then click “Yes, Launch at Login.”

How Nudge is behaving right now matches that scenario exactly (if you count the new config key as the user clicking Yes). With that config key in place, as soon as I manually launch Nudge the first time, it has persistence via SMAppService.

[kevinmcox]

OK I've got a PR up (#516) that mitigates this for now until we have a better understanding of how Apple intends SMAppService to work for tools like Nudge.

[kevinmcox]

Please test this pre-release if you are experiencing this issue: https://github.com/macadmins/nudge/releases/tag/v1.1.13.81503

[natewalck]

After testing with @kevinmcox and a few others on Slack, the following issue still exists:

Case 1

1. Install Nudge config profile
2. Install Nudge_Logger-1.0.1.pkg
3. Install Nudge-1.1.13.81503.pkg
4. Install Nudge_LaunchAgent-1.0.1.pkg
5. Nudge launches immediately
6. 🥳

HOWEVER, if you have 1.1.12 already installed, the following is true:

Case 2

1. Install Nudge config profile
2. Install Nudge_Logger-1.0.1.pkg
3. Install Nudge-1.1.12.whatever.pkg
4. Install Nudge_LaunchAgent-1.0.1.pkg
5. Nudge does not load
6. Install Nudge-1.1.13.81503.pkg
7. Load LaunchAgents or re-install Nudge_LaunchAgent-1.0.1.pkg
8. Nudge does not load

[kevinmcox]

If you deployed 1.1.12 and need to get unstuck, @rickheil actually posted the easiest solution last week in a Slack thread I missed until now.

Either downgrade to 1.1.11 or upgrade to the 1.1.13 pre-release and then run: /usr/bin/sfltool resetbtm

[erikng]

What if we do that with the postinstall?

[kevinmcox]

Sure, we could add a resetbtm script to the package if we don’t think that will cause other non-Nudge related issues.

I’ll PR that in tomorrow if nobody has a better idea and we can still discuss before merging.

[tuxudo]

I think the resetbtm will have negative impacts for end users. During initial troubleshooting I tried it and it cleared out all of the open at login apps from my test user account. The daemons/agents were also cleared out, but they returned after a restart. More testing is needed to determine if this is limited to just my Mac or that's the expected behavior.

[rickheil]

I'm with @tuxudo on that - while it worked fine for my fleet to run resetbtm, I don't know it would necessarily be true universally. Would a release notes mention or similar be sufficient? If you do want to put this in the pkg, I would suggest doing it preinstall so the script can detect if the machine is upgrading from an affected release or not so it is more surgical applying the reset.

[kevinmcox]

Yep, I was thinking preinstall to only run it if 1.1.12 is installed. But sounds like that will be too disruptive in general.

[halwolin]

Does Nudge version 1.1.9.81436 work under Mac OS Sonoma? Or do we need to update all users to the latest version?

From: "Kevin M. Cox" @.> Reply-To: macadmins/nudge @.> Date: Thursday, October 5, 2023 at 9:50 PM To: macadmins/nudge @.> Cc: Subscribed @.> Subject: Re: [macadmins/nudge] Traditional LaunchAgent doesn't load with 1.1.12.81501 (Issue #515)

If you deployed 1.1.12 and need to get unstuck, @rickheilhttps://www.google.com/url?q=https://github.com/rickheil&source=gmail-imap&ust=1697161813000000&usg=AOvVaw1rWoUeKol7fBnl196EMMF9 actually posted the easiest solution last week in a Slack thread I missed until now.

Either downgrade to 1.1.11 or upgrade to the 1.1.13 pre-release and then run: /usr/bin/sfltool resetbtm

— Reply to this email directly, view it on GitHubhttps://www.google.com/url?q=https://github.com/macadmins/nudge/issues/515%23issuecomment-1749878960&source=gmail-imap&ust=1697161813000000&usg=AOvVaw1h7zkrWG27pHgz8ccOAzQX, or unsubscribehttps://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AZ2UVLOP2XDSC3EIYNASHP3X55PVJAVCNFSM6AAAAAA5R34I6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBZHA3TQOJWGA&source=gmail-imap&ust=1697161813000000&usg=AOvVaw2fXsce6276GL_RVJ02bWn7. You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- This email message, including any attachment(s), is intended only for the named recipient(s) and may contain confidential, proprietary or legally privileged information. Unauthorized individuals or entities are not permitted access to this information. Any dissemination, distribution, disclosure, or copying of this information is unauthorized and strictly prohibited. If you have received this message in error, please advise the sender by reply email, and delete this message and any attachments.

[kevinmcox]

@halwolin Version 1.1.9.81436 is over a year old. IMHO I think you should look at updating, but there are several things you'll need to be aware of like the signing certificate change. So make sure to read all the past release notes before you make the jump.

[halwolin]

Thanks. Is the latest version 100% confirmed working with Mac OS Sonoma?

From: "Kevin M. Cox" @.> Reply-To: macadmins/nudge @.> Date: Friday, October 13, 2023 at 11:40 AM To: macadmins/nudge @.> Cc: halwolin @.>, Mention @.***> Subject: Re: [macadmins/nudge] Traditional LaunchAgent doesn't load with 1.1.12.81501 (Issue #515)

@halwolinhttps://www.google.com/url?q=https://github.com/halwolin&source=gmail-imap&ust=1697816432000000&usg=AOvVaw1yBrcjFpWkKP--8PMdQmTc Version 1.1.9.81436 is over a year old. IMHO I think you should look at updating, but there are several things you'll need to be aware of like the signing certificate change. So make sure to read all the past release notes before you make the jump.

— Reply to this email directly, view it on GitHubhttps://www.google.com/url?q=https://github.com/macadmins/nudge/issues/515%23issuecomment-1761723615&source=gmail-imap&ust=1697816432000000&usg=AOvVaw2vakhNIvIl3jm9-fLFSYqw, or unsubscribehttps://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AZ2UVLO4B2ZL4LW5ISDOSHLX7FOG7AVCNFSM6AAAAAA5R34I6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRRG4ZDGNRRGU&source=gmail-imap&ust=1697816432000000&usg=AOvVaw3b4jHy1cQQ_JpBUcjKaRJ4. You are receiving this because you were mentioned.Message ID: @.***>

-- This email message, including any attachment(s), is intended only for the named recipient(s) and may contain confidential, proprietary or legally privileged information. Unauthorized individuals or entities are not permitted access to this information. Any dissemination, distribution, disclosure, or copying of this information is unauthorized and strictly prohibited. If you have received this message in error, please advise the sender by reply email, and delete this message and any attachments.

[erikng]

Yes

[asri-tm]

Hey, we noticed our mac fleet is affected by this. It seems updating to v1.1.13.81503 doesn't automatically fix this?

[kevinmcox]

@asri-tm no you will need to do some manual remediation. Are a member of the free MacAdmins Slack?

[nahooel]

@kevinmcox could you pls share the steps (or the Slack thread) for the manual remediation you mention above? thanks in advance!

[manicmachine]

@nahuelrodriguez-belvo and others finding this thread: the remediation steps are to use the uninstall script pinned in the MacAdmins slack and then reinstall the suite. We had initially tried just installing the fixed version of Nudge overtop our existing installation but that had mixed success. Using the uninstall script has significantly improved our success rate.

[asri-tm]

@nahuelrodriguez-belvo and others finding this thread: the remediation steps are to use the uninstall script pinned in the MacAdmins slack and then reinstall the suite. We had initially tried just installing the fixed version of Nudge overtop our existing installation but that had mixed success. Using the uninstall script has significantly improved our success rate.

Any additional steps that you take before reinstalling nudge-suite? Does the mac need to be restarted before reinstalling?

[kevinmcox]

@BigMacAdmin has also shared that he has found greater success by having a pause between the complete removal and the reinstallation.

My non-reboot resolution to the 1.1.12 issues was to run the uninstall script and then sleep at least 30 second (I think i did 90) and then force reinstall NudgeSuite. That worked to resolve a few hundred problem devices for me

[asri-tm]

@BigMacAdmin has also shared that he has found greater success by having a pause between the complete removal and the reinstallation.

My non-reboot resolution to the 1.1.12 issues was to run the uninstall script and then sleep at least 30 second (I think i did 90) and then force reinstall NudgeSuite. That worked to resolve a few hundred problem devices for me

thanks! In that case, i'll scope in the uninstall policy for the macs that are already on latest then deploy back to all hopefully before the next macOS update release