Macjutsu / super

S.U.P.E.R.M.A.N. optimizes the macOS software update experience.
Apache License 2.0
619 stars 83 forks source link

Local account locks itself #183

Closed dortmund50 closed 2 weeks ago

dortmund50 commented 10 months ago

Hello, We use a macOS configuration profile that blocks the local account after 5 incorrect password entries. Since we rolled out Super on the devices, they seem to lock themselves automatically because the password is entered incorrectly in the background by Super?

Users must enter their password to update; no service account should be used on the device.

Anyone else have this problem? What information is needed?

Thanks

Macjutsu commented 10 months ago

Are your users required to change their local passwords on a regular basis?

As for information... example super.logs showing password failures is where you should start.

coreservices-ron commented 9 months ago

Hi! I can confirm we have a similar behavior with a config profile which requires to change user passwords every 365 days and which locks the account after 5 invalid passwords. Also, FileVault 2 is enabled and users see that they pass through FileVault but are then unable to login to their account. I am currently trying to get a super.log from one of the devices affected and will come back. Kind regards, Ron.

coreservices-ron commented 9 months ago

Hi again! Attached is a log file of an affected device. Strangely enough I can only see that the IBM Notifier dialog was presented once on Dec 14th in order to request the user's password, but then 4 days of silence until the device was restarted today and had the described issue (user account was locked and needed to be unlocked first). I am wondering what happened in the meantime as the 13.6.3 update obviously was not installed. Thank you for any help on this.

super.log

Macjutsu commented 9 months ago

According to that log, you are correct that super asked for auth and then four days later the computer was restarted. In between that time super didn't do anything with the user account. In fact, at no point in that log is there any evidence of super attempting to use any credentials.

For example here are the types of log lines that indicate super is attempt to authenticate: Status: Credentials verified for current user: Status: Saved new --auth-ask-user-to-save-password credentials for current user: Error: User authentication failed.

None of those lines appear in the log, thus the user account being locked has nothing to do with super.

coreservices-ron commented 9 months ago

Thank you very much for your clarification and confirmation! The example lines will help to identify if other computers had issues in combination with the SUPER script. In case I should find any, I will report back again.

coreservices-ron commented 2 months ago

Hi!

I am sorry to reopen this case, which also might be in context with issue #219.

We are still having the issues mentioned. User accounts (auto-locked after 5 failed attempts by password policy) are being locked while the computer is in „sleep“ (lid closed, but apparently on AC and for this reason active) and super is downloading upgrades in the background. As I could research, the failed authentication is not caused by supers check_auth_local_account() function but within the Apple Software Updater call to download the update:

In download_macos_asu() I can see the line echo ' ' | launchctl asuser "${current_user_id}" sudo -u "${current_user_account_name}" softwareupdate --download "${asu_macos_label_target}" --agree-to-license --user "${current_user_account_name}" --stdinpass >> "${ASU_WORKFLOW_LOG}" 2>&1 & for Silicon Macs unter macOS 13+. As the parameter --stdinpass is given, this would mean that in order to download the update, user credentials are given, but an empty password.

In install_macos_asu()the command is used similarly, but this time including the cached password: echo "${auth_local_password}" | sudo -u root softwareupdate --install "${asu_macos_label_target}" --restart --force --no-scan --agree-to-license --user "${auth_local_account}" --stdinpass >> "${ASU_WORKFLOW_LOG}" 2>&1 &

We currently are on 4.0.2, but the respective lines are unchanged in 4.0.3 as well as in 5.0.0beta. So my question is: Is the empty/space password given intentionally for the software download? In asu_workflow_log, we can see the following lines on affected machines: Downloaded: macOS Sonoma 14.6.1 Failed to authenticate

I am unsure if this is the issue or if the account lock was done earlier. But after unlocking the user account (dscl . deletepl /Users/account_name accountPolicyData failedLoginCount) and inserting the retrieved password into download_macos_asu(), the updates instantly worked without further issues.

Sorry for bothering again on this issue, but this causes trouble every time Apple publishes a new update.

Kind regards, Ron.

Macjutsu commented 2 months ago

Thanks for your thorough investigation.

So the goal is to download the update in the background without authentication before bothering the user to restart. To be clear, softwareupdate is designed to do this (as evidence by the default macOS System Settings to do this automatically). However, as you can see, to get it working on-demand via super running as root (as opposed to the end user) requires a messy bunch of hacks.

It would seem that I should probably defer the download workflow until the computer is awake.

coreservices-ron commented 2 months ago

Thank you for your quick reply and clarification!

So actually, you don't see an issue in passing an empty/space stdin string to the softwareupdate --stdinpass parameter? — I also fear that the computers affected simply do not really fall asleep. That's why we see all the IBM Notifier dialogs time out here (we've set a timeout of 10 minutes) and reappear unanswered during the night. For about 5 times they really do appear, after the user account has been locked out (download request) it does not. I guess sudo -u does not work anymore with a locked account.

As I can see from my own experience, MacBook Pros with certain apps installed (Adobe CC, Google [Chrome] Updater, ...) do not hibernate (for long) even when the lid is closed — as long as AC is still connected. I've seen them pulling 40 watts all over the night while being fully charged. Only when power supply is disconnected, they really fall asleep. We've about 500 Macs managed and about 10% are having this use case as we've now found out. And they appear to be affected by the issue.

If it's the lid or (missing) hibernation, that should be doable, but just want to make sure it's not the missing password. — And yes, I've noticed and appreciated the bunch of hacks and special cases you are already handling in super! :-)

Macjutsu commented 2 months ago

Significant testing went into the design of super, and it was determined that the empty password method (for downloads) was the most reliable. This download process is reliable for thousands of computers using super.

Again, downloading macOS updates does NOT require a password. Apple has simply never taken the time to fix the softwareupdate command to reflect this. You can literally try this on your own in Terminal on a Mac that needs updates, enter an equivalent softwareupdate command to download, and when it asks for a password just hit the Enter key with no password.

coreservices-ron commented 1 month ago

Maybe I can illustrate my experiences a bit more based on the attached asu-workflow.log coming from one affected machine. If I am getting things right, super obviously fires one software download command always before notifying the user about the pending update (in order to make sure the correct SWU is downloaded before installation starts). After the initial download was made, subsequent download requests via softwareupdate simply state that the download is processed (again), but obviously find the downloaded files locally and confirm the process as successful. Only if Apple changes the download or deactivates certain versions, files are re-downloaded.

asu-workflow.log

Within the above asu-workflow.log this behavior can be seen for „macOS Sonoma 14.5“ starting June 18: Initial download is being made, preparation at 60% (I guess), done. A subsequent download request follows at 15:21, then it's being installed. No warnings or errors.

But starting with „macOS Sonoma 14.6“ (July 30), the behavior is different: Initial download at 9:01:43, ready at 9:14:50. As the user did not seem to react on the update notification, subsequent download requests are being fired at 9:30:13, 9:46:14, 10:11:15 and so forth. But, after the Downloaded ... line, we see a Failed to authenticate each time (which was not showing for the initial download!). And as far as I can see from the clients, each of these Failed to authenticate lines count against the account lock counter and finally lock out the user after 5 attempts.

The very same situation can be seen for „macOS Sonoma 14.6.1“ last week.

So I would suspect that Apple might have made softwareupdate a bit buggier from 14.5 on which leads to account locks for obsolete download commands. This of course would have to be tested further as soon as there is another update pending. Maybe I will find the time to check this with dscl . readpl /Users/<userName> accountPolicyData failedLoginCount while issuing the command manually on an older system.

I will report back!

coreservices-ron commented 1 month ago

Hi again!

Since there are currently no pending Apple updates for me to test, I did another test on another Silicon system in Terminal which – from my point of view – confirms the suspicion described yesterday:

The test system is on macOS 14.5 currently, so 14.6.1 would be suitable. In administrative context, in Terminal, I issued: sudo -u ron softwareupdate --download "macOS Sonoma 14.6.1-23G93" --agree-to-license --user ron --stdinpass

For the first call, I did not enter my password and simply hit Enter, here the output: Software Update Tool Finding available software Downloading macOS Sonoma 14.6.1 Downloading: 100.00% Downloaded: macOS Sonoma 14.6.1 Failed to authenticate (This would mean that even the first attempt is being counted against account policies as a failed login.)

I repeated the same command (without entering my password) and received the same result (skipping the download itself but still including Failed to authenticate).

For the third time calling the command, I also gave my account password, resulting in: Software Update Tool Finding available software Downloading macOS Sonoma 14.6.1 Downloaded: macOS Sonoma 14.6.1 (No line Failing to authenticate!)

The only difference so far is that my test system does not have local password policies, so I cannot prove that the counter is being incremented. In case you should also need this test, let me know and I will reinstall/downgrade one of the productive machines. But for now I would suggest to pass the password even for the download – as long as it is known to super.

Macjutsu commented 1 month ago

Though I have not had time to confirm this behavior... it would break many super workflows and require significant changes. Many super users leverage MDM authentication or no saved authentication at all... and these workflows would be severely affected if this new behavior is consistent.

Macjutsu commented 1 month ago

Are you restricting macOS updates to administrators only?

coreservices-ron commented 1 month ago

No, not that I knew. Otherwise, updates would not work at all since all cached user passwords are for standard (non-admin) accounts. Also on my test system (actually my own), user „ron“ is a standard account.

I do definitely understand that altering the way downloads are initiated in super might affect other use cases!

Macjutsu commented 1 month ago

So... I just ran super on 14.5 and it successfully downloaded the 14.6.1 update. There must be some other security or user setting that is causing the download to fail when not authenticating.


Tue Aug 13 08:20:37 ASVM-14-5 super[3695]: softwareupdate: Starting macOS Sonoma download workflow, check /Library/Management/super/logs/msu-workflow.log for more detail.
Tue Aug 13 08:20:44 ASVM-14-5 super[3695]: softwareupdate: macOS Sonoma 14.6.1 is downloading...
Tue Aug 13 08:21:17 ASVM-14-5 super[3695]: softwareupdate: macOS Sonoma 14.6.1 download complete, now preparing...
Tue Aug 13 08:29:27 ASVM-14-5 super[3695]: softwareupdate: macOS Sonoma 14.6.1 download and preparation complete.
Macjutsu commented 1 month ago

For good measure... I even ran the line that super uses locally in Terminal... again it worked without the failure you're seeing.

Also to be clear, super only pre-downloads updates if there is an active user... if there is no user (at the login window) then super goes right to the install command so it does authenticate.


Password:
root@ASVM-14-5 ~ # softwareupdate --list
Software Update Tool

Finding available software
Software Update found the following new or updated software:
* Label: SFSymbolsAuto-5.1
    Title: SF Symbols, Version: 5.1, Size: 303685KiB, Recommended: YES, 
* Label: macOS Sonoma 14.6.1-23G93
    Title: macOS Sonoma 14.6.1, Version: 14.6.1, Size: 2140102KiB, Recommended: YES, Action: restart, 
root@ASVM-14-5 ~ # current_user_id=501
root@ASVM-14-5 ~ # current_user_account_name=build
root@ASVM-14-5 ~ # macos_msu_label="macOS Sonoma 14.6.1-23G93"
root@ASVM-14-5 ~ # echo ' ' | launchctl asuser "${current_user_id}" sudo -u "${current_user_account_name}" softwareupdate --download "${macos_msu_label}" --agree-to-license --user "${current_user_account_name}" --stdinpass
Software Update Tool

Finding available software
Downloading macOS Sonoma 14.6.1
Downloading: 60.00%
Macjutsu commented 1 month ago

Well the good news is that I've replicated the issue.... the bad news is that it's definitely the maxFailedAttempts key as part of a Passcode Policy Config Profile.

It appears if that key is deployed then the unauthenticated download does indeed fail.

Sigh... I'll add it to the list of exceptions that I need to work around for janky softwareupdate behavior. No timeline on the fix, as it would entail a lot of changes to the workflow, but I'll leave the v5.0.0 up as the Milestone.

coreservices-ron commented 1 month ago

Dear Kevin,

thank you for your research! I am happy you were able to replicate it though ... I will from my side slightly change the code in the super code we're using (I've done some customization anyway for German dialog localizations) so the cached password will be extracted and given for downloads as well.

Since you were quoting Terminal output two posts above not seeing the Failing to authenticate line: On my test system it came after the Downloaded: macOS Sonoma 14.6.1 part, not during download or before. And since on my test system no account policy was enabled as well as on yours, the maxFailedAttempts did not increase. But on systems having the policy, it obviously does. Well, thanks again Apple for inconsistent behavior and insane command syntax!

¯\_(ツ)_/¯

Macjutsu commented 1 month ago

The good news is that even though I didn't wait to show you the final results... it can work... I tried it again just now.

root@ASVM-14-5 ~ # echo ' ' | launchctl asuser "${current_user_id}" sudo -u "${current_user_account_name}" softwareupdate --download "${macos_msu_label}" --agree-to-license --user "${current_user_account_name}" --stdinpass

Software Update Tool

Finding available software
Downloading macOS Sonoma 14.6.1
Downloading: 100.00%
Downloaded: macOS Sonoma 14.6.1

However, the terrible, awful, horrible, bad news is that I've also discovered that it only works if at no point in the history of the system has anything ever attempted to even engage with pwpolicy. Literally on that same system, right after the download was successful, the mere act of asking pwpolicy what the policy is (again not even setting a policy... just asking for the policy status) engages some mechanism that will then cause the softwareupdate command to fail (even though it's already been downloaded)!!!

Macjutsu commented 1 month ago

Confirmed 14.5 only. 14.4.1 Does not have this problem.

coreservices-ron commented 1 month ago

Thank you for your investigation! Great to know!

Macjutsu commented 4 weeks ago

This is next on my priority list...

coreservices-ron commented 2 weeks ago

Hi Kevin!

Just a brief update: Since macOS 14.7 was released this week, we've had no further lockouts. I indeed was updating and testing the (modified) SUPER release 4.0.2 we were using so far to send the PW along with the user name for downloads, but this was only rolled out on a few test devices. The vast majority of all managed Sonoma Macs (350 of which 10 are test devices) already have the 14.7 update and no user complained about a lockout!

So it may be that Apple revised the behavior in 14.6.1 ...?!

Kind regards, Ron.

Macjutsu commented 2 weeks ago

ugh... of course the results are inconsistent... at this point I'm going to try the unauthenticated version once... then if that fails require authentication.

coreservices-ron commented 2 weeks ago

That was my approach as well: First try one unauthorized download and check user lock status via

loginAllowed=$(pwpolicy authentication-allowed -u "$userName") if [[ "$loginAllowed" != "Policy allows user "* ]]; then if attemptCount=$(dscl . readpl /Users/"$userName" accountPolicyData failedLoginCount); then attemptCount=${attemptCount#failedLoginCount: } # Locked account, failed attempts count else attemptCount=0 # Account is locked, but was unable to read out failed attempts fi else attemptCount=-1 # Account is unlocked fi

In case of a locked out account, I won't try anyway. Otherwise, if the failed attempt count increases after the first download attempt, I will try one more time giving the password. If the counter still increases, give up.

Macjutsu commented 2 weeks ago

My goal is to avoid locking the account entirely by only attempting an unauthenticated download once... then after that assume that authenticated downloads are required for that version of macOS.

The complication comes in where there are so many different authentication and workflow options offered by super that I have to architect a solution for every possilbe permutation and what to do if there are still workflow errors.

Coding = 1% do a thing, 99% don't do the wrong thing.

Macjutsu commented 2 weeks ago

Fixed for local authentication in v5.0.0-beta4... but not for MDM workflows... so keeping it open for now... https://github.com/Macjutsu/super/releases/tag/v5.0.0-beta4

coreservices-ron commented 2 weeks ago

Thank you for your work and clarification(s)!

As for Coding = 1% do a thing, 99% don't do the wrong thing. — can't stress that enough!

A bit offtopic ... but how close is 5.0.0b4 to final? I need to do extensive localization customizations and my customer asks for Sequoia support ...?

And another question for which I did not find the answer quickly enough in the Wiki (sorry!): If Major Software Updates are postponed for 90 days in a config profile, will super (4.0.2) download and install a macOS 15 Upgrade if requested via --install-macos-major-upgrades --install-macos-major-version-target=15 today?

Cheers, Ron.

Macjutsu commented 2 weeks ago

The final big push for v5.0.0 completion is almost entirely MDM workflow stuff. As far as changes to set_display_strings_language() in v5.0.0... I don't have any plans to make anymore changes. That doesn't mean I won't find bugs in there, just that I don't think there should be anymore big changes.

I have never tested v4 against macOS Sequioa... I have heard reports that upgrades from macOS 14 work... but I know for a fact that older OSes will not work when upgrading to macOS 15 and that the restart validation on macOS 15 might mistake security updates as system updates.

Macjutsu commented 2 weeks ago

Closing this issue with the release of https://github.com/Macjutsu/super/releases/tag/v5.0.0-beta5

So because the macOS download authentication failure is inconsistent, super still attempts an unauthenticated download first. If an authentication failure is detected, all future download attempts for that version of macOS will be authenticated (or failover if saved authentication isn't availalbe or valid). As long as you aren't constantly using --reset-super, then the download workflow should not longer lock out any user accounts even if there are password policies.

coreservices-ron commented 2 weeks ago

Thank you very much! Will keep you updated.