search

grahampugh / erase-install

A script that automates downloading macOS installers, and optionally erasing or upgrading macOS in a single process. Watch the video!
https://grahamrpugh.com/2023/05/14/macaduk-presentation-eraseinstall.html
Apache License 2.0
837 stars 131 forks source link

Script fails if invalid installer is already cached regardless of --overwrite, --replace_invalid --update options #316

Closed Rorzzer closed 1 year ago

Rorzzer commented 1 year ago

Describe the bug

Firstly, caching macOS 13 via the script seems to grab an invalid build OR the script incorrectly identifies a cached build as invalid then fails. The error message says to use the --overwrite option even when using the --overwrite option

Our Caching command - that I believe is working but the installer is considered invalid later on?

/Library/Management/erase-install/erase-install.sh --os=13 --overwrite --move --force-curl

Our install command - that marks the cached installer as invalid and says to run with the --overwrite option

/Library/Management/erase-install/erase-install.sh --reinstall --os=13 --overwrite --depnotify --cleanup-after-use --check-power --force-curl --power-wait-limit 180

other commands I've tried that give the exact same error message,

/Library/Management/erase-install/erase-install.sh --reinstall --os=13 --replace_invalid --depnotify --cleanup-after-use --check-power --force-curl --power-wait-limit 180

/Library/Management/erase-install/erase-install.sh --reinstall --os=13 --update --depnotify --cleanup-after-use --check-power --force-curl --power-wait-limit 180

To Reproduce

Expected behavior A clear and concise description of what you expected to happen. First command should grab a valid installer and cache it. Second command should see the valid installer and install it without an error message.

Code/log output Please supply the full command used, and if applicable, add full output from Terminal or from /var/log/erase-install.log. Either upload the log, or paste the output in a code block (triple backticks at the start and end of the code block please!).

Caching Log

Executing Policy Cache macOS Ventura 13 Intel
--
Downloading erase-install-depnotify-27.1.pkg...
Downloading https://apse2-jcds.services.jamfcloud.com//download/fbef3a7a309849f885e3bb31e41434c1/erase-install-depnotify-27.1.pkg?token=e7ffc84da7af410dba9362388820448508r1vvevm5uqle8emewz0y83tvssjukd...
Verifying package integrity...
Installing erase-install-depnotify-27.1.pkg...
Successfully installed erase-install-depnotify-27.1.pkg.
Running command /Library/Management/erase-install/erase-install.sh --os=13 --overwrite --move --force-curl...
Result of command:id: : no such user Unexpected character n at line 1 Print: Entry, "AppleLanguages:0", Does Not Exist     [erase-install] v27.1 script execution started: Wed Nov 23 15:54:44 AEDT 2022    [erase-install] Caffeinating this script (pid=639)    [erase-install] Looking for existing installer app or pkg    [find_existing_installer] No valid installer found.    [erase-install] Existing installer does not match requested OS, so replacing...    [overwrite_existing_installer] Overwrite option selected. Deleting existing version.    [erase-install] Running on architecture i386    [check_installinstallmacos] force-curl selected so installinstallmacos.py will be overwritten    [download_installinstallmacos] Downloading installinstallmacos.py...    [check_installinstallmacos] installinstallmacos.py is in /Library/Management/erase-install    [check_python] Relocatable Python is installed in /Library/Management/erase-install    [run_installinstallmacos] Checking that selected OS 13 is available     [run_installinstallmacos] This command is now being run:     installinstallmacos.py --workdir /Library/Management/erase-install --ignore-cache --raw --os 13   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                  Dload  Upload   Total   Spent    Left  Speed    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0 100 1303k  100 1303k    0     0  5003k      0 --:--:-- --:--:-- --:--:-- 5131k   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                  Dload  Upload   Total   Spent    Left  Speed    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0 100  5044  100  5044    0     0    99k      0 --:--:-- --:--:-- --:--:--  117k   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                  Dload  Upload   Total   Spent    Left  Speed    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0 100   181  100   181    0     0   3550      0 --:--:-- --:--:-- --:--:--  4209   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                  Dload  Upload   Total   Spent    Left  Speed    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  19 3369k   19  655k    0     0  4286k      0 --:--:-- --:--:-- --:--:-- 4458k 100 3369k  100 3369k    0     0  5904k      0 --:--:-- --:--:-- --:--:-- 5963k   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                  Dload  Upload   Total   Spent    Left  Speed    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0 100 3277k  100 3277k    0     0  5885k      0 --:--:-- --:--:-- --:--:-- 5969k   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                  Dload  Upload   Total   Spent    Left  Speed    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0   0 11.3G    0 6191k    0     0  6147k      0  0:32:10  0:00:01  0:32:09 6185k   0 11.3G    0 12.4M    0     0  6326k      0  0:31:15  

[skipping download details]

      0  1:31:31  1:31:29  0:00:02 2063k  99 11.3G   99 11.3G    0     0  2160k      0  1:31:31  1:31:30  0:00:01 2073k  99 11.3G   99 11.3G    0     0  2160k      0  1:31:31  1:31:31 --:--:-- 2086k 100 11.3G  100 11.3G    0     0  2160k      0  1:31:31  1:31:31 --:--:-- 2085k installer: Package name is macOS Ventura installer: Installing at base path /private/tmp/dmg.8A5ou3 installer: The install was successful.  installinstallmacos.py - get macOS installers from the Apple software catalog  This Mac: Model Identifier : MacBookPro15,1  Bridge ID        : J680AP Board ID         : Mac-937A206F2EE63C01 OS Version       : 12.6.1 Build ID         : 21G217   #  ProductID       Version    Build    Post Date   Title                            1  061-26578       10.14.5    18F2059  2019-10-14  macOS Mojave                     2  061-26589       10.14.6    18G103   2019-10-14  macOS Mojave                     3  041-91758       10.13.6    17G66    2019-10-19  macOS High Sierra                4  041-88800       10.14.4    18E2034  2019-10-23  macOS Mojave                     5  041-90855       10.13.5    17F66a   2019-10-23  Install macOS High Sierra Beta   6  061-86291       10.15.3    19D2064  2020-03-23  macOS Catalina                   7  001-04366       10.15.4    19E2269  2020-05-04  macOS Catalina                   8  001-15219       10.15.5    19F2200  2020-06-15  macOS Catalina                   9  001-36735       10.15.6    19G2006  2020-08-06  macOS Catalina                  10  001-36801       10.15.6    19G2021  2020-08-12  macOS Catalina                  11  001-51042       10.15.7    19H2     2020-09-24  macOS Catalina                  12  001-57224       10.15.7    19H4     2020-10-27  macOS Catalina                  13  001-68446       10.15.7    19H15    2020-11-11  macOS Catalina                  14  071-78704       11.5.2     20G95    2021-08-18  macOS Big Sur                   15  002-23589       11.6.1     20G224   2021-12-01  macOS Big Sur                   16  002-42341       11.6.2     20G314   2022-01-14  macOS Big Sur                   17  002-57023       11.6.3     20G415   2022-01-26  macOS Big Sur                   18  002-65695       11.6.4     20G417   2022-02-17  macOS Big Sur                   19  002-77154       11.6.5     20G527   2022-04-11  macOS Big Sur                   20  012-08272       11.6.6     20G624   2022-05-24  macOS Big Sur                   21  012-40387       11.6.8     20G730   2022-07-28  macOS Big Sur                   22  012-51693       12.5.1     21G83    2022-08-24  macOS Monterey                  23  012-40494       12.6       21G115   2022-09-20  macOS Monterey                  24  012-38280       11.7       20G817   2022-09-20  macOS Big Sur                   25  012-92138       13.0       22A380   2022-10-24  macOS Ventura                   26  012-90254       12.6.1     21G217   2022-10-24  macOS Monterey                  27  012-90253       11.7.1     20G918   2022-10-24  macOS Big Sur                   28  012-93766       13.0.1     22A400   2022-11-09  macOS Ventura                    Build 22A400 selected. Downloading #28...  Making empty sparseimage... ********************************************************* *** Working around a very dumb Apple bug in a package *** *** postinstall script that fails to correctly target *** *** the Install macOS.app when installed to a volume  *** *** other than the current boot volume.               *** ***       Please file feedback with Apple!            *** ********************************************************* Product downloaded and installed to /Library/Management/erase-install/Install_macOS_13.0.1-22A400.sparseimage    [run_installinstallmacos] Mounting sparse disk image to identify installer app.    [erase-install] Installer is at: /Volumes/Install_macOS_13.0.1-22A400/Applications/Install macOS Ventura.app    [erase-install] Invoking --move option    [move_to_applications_folder] Moving installer to /Applications folder    [move_to_applications_folder] Mounted installer will be unmounted: /Volumes/Install_macOS_13.0.1-22A400/Applications/Install macOS Ventura.app Volume Install_macOS_13.0.1-22A400 on disk2s2 force-unmounted    [move_to_applications_folder] Installer moved to /Applications folder    [erase-install] Closing jamfHelper download message (language=en)     [erase-install] Cleaning working directory '/Library/Management/erase-install/content'     [erase-install] attempting to terminate the 'caffeinate' process - Termination message indicates success    [erase-install] ERROR: 'caffeinate' could not be killed     [erase-install] attempting to terminate the 'caffeinate' process - Termination message indicates success    [erase-install] ERROR: 'caffeinate' could not be killed     [finish] Script exit code: 0

Install log

Executing Policy Upgrade to macOS Ventura 13 - Intel
--
[STEP 2 of 5]
Downloading erase-install-depnotify-27.1.pkg...
Downloading https://apse2-jcds.services.jamfcloud.com//download/fbef3a7a309849f885e3bb31e41434c1/erase-install-depnotify-27.1.pkg?token=e7ffc84da7af410dba9362388820448508r1vvevm5uqle8emewz0y83tvssjukd...
Verifying package integrity...
Installing erase-install-depnotify-27.1.pkg...
Successfully installed erase-install-depnotify-27.1.pkg.
[STEP 3 of 5]
[STEP 4 of 5]
Running command /Library/Management/erase-install/erase-install.sh --reinstall --os=13 --overwrite --depnotify --cleanup-after-use --check-power --force-curl --power-wait-limit 180...
Result of command:[erase-install] v27.1 script execution started: Wed Nov 23 19:16:28 AEDT 2022    [erase-install] Caffeinating this script (pid=2539)    [check_free_space] OK - 371 GB free/purgeable disk space detected    [check_power_status] OK - AC power detected    [erase-install] Looking for existing installer app or pkg    [find_existing_installer] Installer sparse image found at /Library/Management/erase-install/Install_macOS_13.0.1-22A400.sparseimage.    [check_installer_is_valid] Checking validity of .    [check_installer_is_valid] Using DTSDKBuild value from Info.plist    [check_installer_is_valid] Installer Info.plist could not be found!    [check_installer_is_valid] Build of existing installer could not be found, so it is assumed to be invalid.    [erase-install] ERROR: Invalid installer is present. Run with --overwrite option to ensure that a valid installer is obtained.     [erase-install] attempting to terminate the 'caffeinate' process - Termination message indicates success /Library/Management/erase-install/erase-install.sh: line 1046:  2559 Terminated: 15          /usr/bin/caffeinate -dimsu -w $     [finish] Script exit code: 1

Screenshots If applicable, add screenshots to help explain your problem.

Screenshot 2022-11-24 at 10 40 08 am Screenshot 2022-11-24 at 10 37 33 am Screenshot 2022-11-24 at 10 33 42 am

Environment (please complete the following information):

Additional context Add any other context about the problem here.

Rorzzer commented 1 year ago

Previously it seemed to be stumbling on the old Big Sur installer that was in the applications folder but more recently it is finding the cached Ventura installer but failing to get the details from it then not obeying the --overwrite command and simply failing.

I've also confirmed that the Ventura installer is in the Applications folder but the script doesn't seem to be finding that.

Screenshot 2022-11-24 at 11 10 44 am
Rorzzer commented 1 year ago

This is the most recent error message from a device that has the installer in the Applications folder but seems to have a broken/invalid installer in a sparesimage? Is it just a matter of deleting the spareimage? Is it a bug that the script is not obeying the --overwrite option?

Screenshot 2022-11-24 at 11 16 47 am
HeywoodJa commented 1 year ago

I have this issue too.

grahampugh commented 1 year ago

Most times that people have brought this up, it's because there is some security software which is preventing the disk image (sparseimage) from mounting. This prevents the downloaded installer from being checked.

You also have some mistakes in your commands, which are not the cause of your error, but should be addressed nonetheless, e.g.:

  1. You should not use the --force-curl option when using the installer package, because package comes with the correct version of installinstallmacos.pay already.

  2. You are separately caching the installer but then your second command has the --overwrite flag, which will overwrite any cached installer!

Rorzzer commented 1 year ago

I was only testing with the --overwrite flag, usually we would use --update but I've found that regardless of the flag the same behaviour is observed. I also tried this on a device without any anti-virus or security software installed and found the same result.

Rorzzer commented 1 year ago

We've moved away from caching installers until this is resolved. The behaviour when an installer is already present is unpredictable and fails entirely if multiple installers exist in the Applications folder. For now we are back to --update --move and if it fails we forcibly remove all installers with another policy then try again.

mtucker299 commented 1 year ago

@Rorzzer,

I am seeing the same behavior on my Macs as well. I have also tried --update and also --overwrite to no avail.

In looking in the /Library/Management/erase-install/ folder I see a folder called "content" to which no one but system has access to. This folder doesn't exist on other successfully upgraded systems but does on all failed ones. I granted permissions and deleted the folder and it seems that the download process creates it immediately.

Once I deleted this folder on a system, the script seems to be running.

For reference, here is my script with flags /Library/Management/erase-install/erase-install.sh --reinstall --os=13 --current-user --min-drive-space=60 --check-power --depnotify --overwrite

Hope this helps.

grahampugh commented 1 year ago

The content folder is where installinstallmacos.py stores the downloaded content from Apple's software update servers.

We don't have a really good way to detect a corrupt downloaded installer, although 27.1 should be re-downloading if it doesn't get a version string from the installer. If that's not working for some reason (I will check this again when I'm on a network that's fast enough), there are various built-in ways to workaround all this:

  1. Use the --move flag with your caching policy so that you end up with an installer in the Applications folder rather than the sparseimage (which is the default item created by installinstallmacos.py).
  2. Use the --cleanup-after-use flag to delete the /Library/Contents/erase-install folder and all its contents on the next reboot (this would not delete an installer that had been moved to the Applications folder).

Alternatively, use the --pkg option which creates a package instead of a sparseimage, for those of you who have computers that are struggling to read the contents of the sparseimage properly.

Or even use the --fetch-full-installer which uses Apple's built-in system for obtaining the installed (has one advantage that it can get the installer from a caching server if you have one).

mtucker299 commented 1 year ago

Thanks for response.

I tried the --cleanup-after-use to no avail, but it could be because I don't have the /Library/Content/ folder. My erase-install installed to /Library/Management/erase-install/ so maybe that's why the flag didn't work.

I will add the --move flag to see how that works.

grahampugh commented 1 year ago

Huh? I guess you misunderstood.

The normal path for erase-install.sh installation is /Library/Management/erase-install. installinstallmacos.py is also installed in that folder. Therefore, it stores its working files in /Library/Management/erase-install/content.

mtucker299 commented 1 year ago

It seems I did. The cleanup process doesn’t appear to remove the “content” folder. I’ll try it again.

Thanks,

Matt

mtucker299 commented 1 year ago
  1. Use the --cleanup-after-use flag to delete the /Library/Contents/erase-install folder and all its contents on the next reboot (this would not delete an installer that had been moved to the Applications folder).

This is what I was misunderstanding I thought you literally meant it was deleting the folder listed above, not the content folder under erase-install.

Am I still off target?

grahampugh commented 1 year ago

Huh, sorry, if I wrote that I must have been tired!

It should say: Use the --cleanup-after-use flag to delete the /Library/Management/erase-install folder and all its contents on the next reboot

mtucker299 commented 1 year ago

Thanks for taking the time to follow up! I think I'm understanding now.

grahampugh commented 1 year ago

This issue was fixed in 27.2. The current version is 28.1.