grahampugh / erase-install

A script that automates downloading macOS installers, and optionally erasing or upgrading macOS in a single process. Watch the video!
https://grahamrpugh.com/2023/05/14/macaduk-presentation-eraseinstall.html
Apache License 2.0
834 stars 131 forks source link

[BUG] Fetch Full Installer Flag failing (& overwriting existing version despite the Update flag) #468

Closed Rorzzer closed 9 months ago

Rorzzer commented 10 months ago

Describe the bug --ffi fails with error "2023-11-15 12:53:45 | v31.0 | [run_fetch_full_installer] Could not obtain installer information using softwareupdate --list-full-installers. Cannot continue."

Also the --update flag is not honoured and the script reports "2023-11-15 12:53:38 | v31.0 | [overwrite_existing_installer] Overwrite option selected. Deleting existing version."

To Reproduce

Expected behavior A clear and concise description of what you expected to happen. Device to update to latest macOS14 version from Apple servers, (ideally via our content cache but that is not script related).

Code/log output Please supply the full command used, and if applicable, add full output from Terminal or from /var/log/erase-install.log. Don't post a partial log - I need the whole thing. Either upload the log as a file, or paste the output in a code block like this:

Executing Policy Upgrade to macOS 14 Sonoma - Head Office
Mounting JAMF File Share
Verifying package integrity...
Installing erase-install-31.0.pkg...
Successfully installed erase-install-31.0.pkg.
Running script erase-install-launcher...
Script exit code: 1
Script result: [erase-install-launcher] Starting script "/Library/Management/erase-install/erase-install.sh" --reinstall --update --rebootdelay=120 --power-wait-limit=180 --ffi --os=14
2023-11-15 14:42:38 | v31.0 | [log_rotate] Finished rotating logs in /Library/Management/erase-install/log

2023-11-15 14:42:38 | v31.0 | [erase-install] v31.0 script execution started: Wed 15 Nov 2023 14:42:38 AEDT

2023-11-15 14:42:38 | v31.0 | [erase-install] Arguments provided: --reinstall --update --rebootdelay=120 --power-wait-limit=180 --ffi --os=14
2023-11-15 14:42:38 | v31.0 | [erase-install] System version: 14.0 (Build: 23A344)
2023-11-15 14:42:38 | v31.0 | [check_for_swiftdialog_app] swiftDialog is installed (/Library/Application Support/Dialog/Dialog.app)
2023-11-15 14:42:38 | v31.0 | [check_for_swiftdialog_app] Creating dialog log (/var/tmp/dialog.dTZ)...
2023-11-15 14:42:38 | v31.0 | [erase-install] Caffeinating this script (pid=30311)
2023-11-15 14:42:38 | v31.0 | [check_free_space] OK - 194 GB free/purgeable disk space detected
2023-11-15 14:42:38 | v31.0 | [erase-install] Looking for existing installer app or pkg
2023-11-15 14:42:38 | v31.0 | [find_existing_installer] No valid installer found.
2023-11-15 14:42:38 | v31.0 | [erase-install] Existing installer does not match requested OS (14), so replacing...
2023-11-15 14:42:38 | v31.0 | [overwrite_existing_installer] Overwrite option selected. Deleting existing version.
overwrite_existing_installer:2: no matches found: /Volumes/*macOS*
2023-11-15 14:42:38 | v31.0 | [erase-install] Running on architecture arm64
2023-11-15 14:42:38 | v31.0 | [get_user_details] ask for user credentials (attempt 1/5)
2023-11-15 14:42:38 | v31.0 | [get_default_dialog_args] Invoking utility dialog
yes powellr is a member of everyone
2023-11-15 14:43:38 | v31.0 | [get_user_details] powellr is a Volume Owner
2023-11-15 14:43:38 | v31.0 | [check_password] Success: the password entered is the correct login password for powellr.
2023-11-15 14:43:38 | v31.0 | [get_default_dialog_args] Invoking utility dialog
2023-11-15 14:43:39 | v31.0 | [run_fetch_full_installer] Could not obtain installer information using softwareupdate --list-full-installers. Cannot continue.

2023-11-15 14:43:39 | v31.0 | [erase-install] attempting to terminate the 'caffeinate' process - Termination message indicates success

2023-11-15 14:43:39 | v31.0 | [finish] sending quit message to dialog (/var/tmp/dialog.dTZ)
2023-11-15 14:43:39 | v31.0 | [finish] Script exit code: 1
[erase-install-launcher] Exit (1)

Error running script: return code was 1.

Screenshots If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

Additional context Add any other context about the problem here.

Rorzzer commented 10 months ago

Confirming that the softwareupdate --list-full-installers command does work outside of the script.

powellr@Rorys-MacBook-Pro-2 ~ % softwareupdate --list-full-installers Finding available software Software Update found the following full installers:

Rorzzer commented 10 months ago

Some other command results below,

sudo /Library/Management/erase-install/erase-install.sh --list

powellr@Rorys-MacBook-Pro-2 ~ % sudo /Library/Management/erase-install/erase-install.sh --list      
2023-11-15 14:58:01 | v31.0 | [log_rotate] Finished rotating logs in /Library/Management/erase-install/log

2023-11-15 14:58:01 | v31.0 | [erase-install] v31.0 script execution started: Wed 15 Nov 2023 14:58:01 AEDT

2023-11-15 14:58:01 | v31.0 | [erase-install] Arguments provided: --list
2023-11-15 14:58:01 | v31.0 | [erase-install] System version: 14.0 (Build: 23A344)
2023-11-15 14:58:01 | v31.0 | [check_for_swiftdialog_app] swiftDialog is installed (/Library/Application Support/Dialog/Dialog.app)
2023-11-15 14:58:01 | v31.0 | [check_for_swiftdialog_app] Creating dialog log (/var/tmp/dialog.7zC)...
2023-11-15 14:58:01 | v31.0 | [check_for_mist] mist is installed (/usr/local/bin/mist)
2023-11-15 14:58:01 | v31.0 | [get_mist_list] Non-default catalog selected (darwin version 23)
┌──────────────────┐
│ INPUT VALIDATION │
└──────────────────┘
  ├─ Search only for latest (first) result will be 'false'...
  ├─ Include betas in search results will be 'false'...
  ├─ Only include compatible installers will be 'true'...
  ├─ Export path will be '/Library/Management/erase-install/mist-list.json'...
  ├─ Export path file extension is valid...
  ├─ Output type will be 'ascii'...
┌────────┐
│ SEARCH │
└────────┘
  ├─ Searching for macOS Installer versions...
  ├─ Exported list as JSON: '/Library/Management/erase-install/mist-list.json'
  └─ Found 15 macOS Installer(s) available for download

┌────────────┬────────────────┬─────────┬────────┬──────────┬────────────┬────────────┐
│ IDENTIFIER │ NAME           │ VERSION │ BUILD  │ SIZE     │ DATE       │ COMPATIBLE │
├────────────┼────────────────┼─────────┼────────┼──────────┼────────────┼────────────┤
│ 042-89627  │ macOS Sonoma   │ 14.1.1  │ 23B81  │ 12.92 GB │ 2023-11-08 │ True       │
│ 042-86434  │ macOS Sonoma   │ 14.1    │ 23B74  │ 12.92 GB │ 2023-10-26 │ True       │
│ 042-58988  │ macOS Sonoma   │ 14.0    │ 23A344 │ 12.87 GB │ 2023-09-27 │ True       │
│ 042-95459  │ macOS Ventura  │ 13.6.2  │ 22G320 │ 10.48 GB │ 2023-11-08 │ True       │
│ 042-82952  │ macOS Ventura  │ 13.6.1  │ 22G313 │ 11.95 GB │ 2023-10-26 │ True       │
│ 042-55926  │ macOS Ventura  │ 13.6    │ 22G120 │ 11.95 GB │ 2023-09-22 │ True       │
│ 042-43677  │ macOS Ventura  │ 13.5.2  │ 22G91  │ 11.95 GB │ 2023-09-08 │ True       │
│ 042-25643  │ macOS Ventura  │ 13.5.1  │ 22G90  │ 11.95 GB │ 2023-08-18 │ True       │
│ 032-69593  │ macOS Ventura  │ 13.5    │ 22G74  │ 11.95 GB │ 2023-07-25 │ True       │
│ 042-01917  │ macOS Ventura  │ 13.4.1  │ 22F82  │ 11.80 GB │ 2023-06-29 │ True       │
│ 042-82938  │ macOS Monterey │ 12.7.1  │ 21G920 │ 12.41 GB │ 2023-10-26 │ True       │
│ 042-55586  │ macOS Monterey │ 12.7    │ 21G816 │ 12.40 GB │ 2023-09-22 │ True       │
│ 042-45268  │ macOS Monterey │ 12.6.9  │ 21G726 │ 12.41 GB │ 2023-09-12 │ True       │
│ 042-15015  │ macOS Monterey │ 12.6.8  │ 21G725 │ 12.42 GB │ 2023-07-25 │ True       │
│ 032-97690  │ macOS Monterey │ 12.6.7  │ 21G651 │ 12.41 GB │ 2023-06-29 │ True       │
└────────────┴────────────────┴─────────┴────────┴──────────┴────────────┴────────────┘

2023-11-15 14:58:06 | v31.0 | [finish] Script exit code: 0

sudo /Library/Management/erase-install/erase-install.sh --list --ffi

powellr@Rorys-MacBook-Pro-2 ~ % sudo /Library/Management/erase-install/erase-install.sh --list --ffi
2023-11-15 15:00:01 | v31.0 | [log_rotate] Finished rotating logs in /Library/Management/erase-install/log

2023-11-15 15:00:01 | v31.0 | [erase-install] v31.0 script execution started: Wed 15 Nov 2023 15:00:01 AEDT

2023-11-15 15:00:01 | v31.0 | [erase-install] Arguments provided: --list --ffi
2023-11-15 15:00:01 | v31.0 | [erase-install] System version: 14.0 (Build: 23A344)
2023-11-15 15:00:02 | v31.0 | [check_for_swiftdialog_app] swiftDialog is installed (/Library/Application Support/Dialog/Dialog.app)
2023-11-15 15:00:02 | v31.0 | [check_for_swiftdialog_app] Creating dialog log (/var/tmp/dialog.8T3)...
cat: /Library/Management/erase-install/ffi-list-full-installers.txt: No such file or directory

2023-11-15 15:00:02 | v31.0 | [finish] Script exit code: 0
grahampugh commented 10 months ago

The overwrite section is just the part where a new installer is downloaded, and being induced because there is no installer currently on the system.

Something prevented the computer from obtaining a list of installers using the softwareupdate command. Check your network. Try running the command manually on the affected computer:

softwareupdate --list-full-installers

Rorzzer commented 10 months ago

Hey @grahampugh, see comments above. Commands all work fine separately but not via the script.

kMikaZu commented 9 months ago

I can confirm this behaviour. The script checks for a download and then just quits. Nothing changed to the config, network is fine. Also tested with version 31.0 and the --overwrite parameter but that doesn't help.

Log:

[STEP 1 of 6]
Executing Policy 01-Upgrade to MINOR macOS version UPDATE_PRODUCTIE-LABO: 13.6.2
[STEP 2 of 6]
Mounting CACHING-SERVER
Verifying package integrity...
Installing erase-install-31.0.pkg...
Successfully installed erase-install-31.0.pkg.
[STEP 3 of 6]
Running script Upgrade MINOR macOS (Nudge + EraseInstall)...
Script exit code: 0
Script result: killing the running nudge window
Nudge launch agent has been unloaded
Erase-install script active, installing macOS - 13.6.2.
2023-11-22 15:46:13 | v31.0 | [log_rotate] Finished rotating logs in /Library/Management/erase-install/log

2023-11-22 15:46:13 | v31.0 | [erase-install] v31.0 script execution started: Wed Nov 22 15:46:13 CET 2023

2023-11-22 15:46:13 | v31.0 | [erase-install] Arguments provided: --power-wait-limit 180 --check-power --cleanup-after-use --current-user --reinstall --fs --version=13.6.2 --min-drive-space 20 --overwrite
2023-11-22 15:46:13 | v31.0 | [erase-install] System version: 13.6.1 (Build: 22G313)
2023-11-22 15:46:14 | v31.0 | [check_for_swiftdialog_app] swiftDialog v2.3.2.4726 is installed (/Library/Application Support/Dialog/Dialog.app)
2023-11-22 15:46:14 | v31.0 | [check_for_swiftdialog_app] Creating dialog log (/var/tmp/dialog.mjV)...
2023-11-22 15:46:14 | v31.0 | [erase-install] Caffeinating this script (pid=3462)
2023-11-22 15:46:14 | v31.0 | [check_free_space] OK - 202 GB free/purgeable disk space detected
2023-11-22 15:46:14 | v31.0 | [erase-install] Looking for existing installer app or pkg
2023-11-22 15:46:14 | v31.0 | [find_existing_installer] No valid installer found.
2023-11-22 15:46:14 | v31.0 | [erase-install] Running on architecture arm64
2023-11-22 15:46:14 | v31.0 | [get_user_details] ask for user credentials (attempt 1/5)
2023-11-22 15:46:14 | v31.0 | [get_default_dialog_args] Invoking utility dialog
yes user is a member of everyone
2023-11-22 15:46:21 | v31.0 | [get_user_details] user@user.be is a Volume Owner
2023-11-22 15:46:21 | v31.0 | [check_password] Success: the password entered is the correct login password for user@user.be.
2023-11-22 15:46:21 | v31.0 | [check_power_status] OK - AC power detected
2023-11-22 15:46:21 | v31.0 | [get_default_dialog_args] Invoking fullscreen dialog
2023-11-22 15:46:21 | v31.0 | [check_for_mist] mist is installed (/usr/local/bin/mist)
2023-11-22 15:46:21 | v31.0 | [get_mist_list] Non-default catalog selected (darwin version 22)
┌──────────────────┐
│ INPUT VALIDATION │
└──────────────────┘
  ├─ Search only for latest (first) result will be 'false'...
  ├─ Include betas in search results will be 'false'...
  ├─ Only include compatible installers will be 'true'...
  ├─ Export path will be '/Library/Management/erase-install/mist-list.json'...
  ├─ Export path file extension is valid...
  ├─ Output type will be 'ascii'...
┌────────┐
│ SEARCH │
└────────┘
  ├─ Searching for macOS Installer versions...
  ├─ Exported list as JSON: '/Library/Management/erase-install/mist-list.json'
  └─ Found 17 macOS Installer(s) available for download

[0;0m┌────────────┬────────────────┬─────────┬─────────┬──────────┬────────────┬────────────┐
│ IDENTIFIER │ NAME           │ VERSION │ BUILD   │ SIZE     │ DATE       │ COMPATIBLE │
├────────────┼────────────────┼─────────┼─────────┼──────────┼────────────┼────────────┤
[0;0m[0;0m│ [0;0m042-89627 [0;0m │ [0;0mmacOS Sonoma  [0;0m │ [0;0m14.1.1 [0;0m │ [0;0m23B81  [0;0m │ [0;0m12.92 GB[0;0m │ [0;0m2023-11-16[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m042-86434 [0;0m │ [0;0mmacOS Sonoma  [0;0m │ [0;0m14.1   [0;0m │ [0;0m23B74  [0;0m │ [0;0m12.92 GB[0;0m │ [0;0m2023-10-25[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m042-58988 [0;0m │ [0;0mmacOS Sonoma  [0;0m │ [0;0m14.0   [0;0m │ [0;0m23A344 [0;0m │ [0;0m12.87 GB[0;0m │ [0;0m2023-09-26[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m042-82952 [0;0m │ [0;0mmacOS Ventura [0;0m │ [0;0m13.6.1 [0;0m │ [0;0m22G313 [0;0m │ [0;0m11.95 GB[0;0m │ [0;0m2023-11-16[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m042-55926 [0;0m │ [0;0mmacOS Ventura [0;0m │ [0;0m13.6   [0;0m │ [0;0m22G120 [0;0m │ [0;0m11.95 GB[0;0m │ [0;0m2023-09-21[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m042-43677 [0;0m │ [0;0mmacOS Ventura [0;0m │ [0;0m13.5.2 [0;0m │ [0;0m22G91  [0;0m │ [0;0m11.95 GB[0;0m │ [0;0m2023-09-07[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m042-25643 [0;0m │ [0;0mmacOS Ventura [0;0m │ [0;0m13.5.1 [0;0m │ [0;0m22G90  [0;0m │ [0;0m11.95 GB[0;0m │ [0;0m2023-08-17[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m032-69593 [0;0m │ [0;0mmacOS Ventura [0;0m │ [0;0m13.5   [0;0m │ [0;0m22G74  [0;0m │ [0;0m11.95 GB[0;0m │ [0;0m2023-07-24[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m042-01917 [0;0m │ [0;0mmacOS Ventura [0;0m │ [0;0m13.4.1 [0;0m │ [0;0m22F82  [0;0m │ [0;0m11.80 GB[0;0m │ [0;0m2023-06-28[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m042-82938 [0;0m │ [0;0mmacOS Monterey[0;0m │ [0;0m12.7.1 [0;0m │ [0;0m21G920 [0;0m │ [0;0m12.41 GB[0;0m │ [0;0m2023-11-16[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m042-55586 [0;0m │ [0;0mmacOS Monterey[0;0m │ [0;0m12.7   [0;0m │ [0;0m21G816 [0;0m │ [0;0m12.40 GB[0;0m │ [0;0m2023-09-21[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m042-45268 [0;0m │ [0;0mmacOS Monterey[0;0m │ [0;0m12.6.9 [0;0m │ [0;0m21G726 [0;0m │ [0;0m12.41 GB[0;0m │ [0;0m2023-09-11[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m042-15015 [0;0m │ [0;0mmacOS Monterey[0;0m │ [0;0m12.6.8 [0;0m │ [0;0m21G725 [0;0m │ [0;0m12.42 GB[0;0m │ [0;0m2023-07-24[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m032-97690 [0;0m │ [0;0mmacOS Monterey[0;0m │ [0;0m12.6.7 [0;0m │ [0;0m21G651 [0;0m │ [0;0m12.41 GB[0;0m │ [0;0m2023-06-28[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m042-45246 [0;0m │ [0;0mmacOS Big Sur [0;0m │ [0;0m11.7.10[0;0m │ [0;0m20G1427[0;0m │ [0;0m12.42 GB[0;0m │ [0;0m2023-09-11[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m042-14707 [0;0m │ [0;0mmacOS Big Sur [0;0m │ [0;0m11.7.9 [0;0m │ [0;0m20G1426[0;0m │ [0;0m12.42 GB[0;0m │ [0;0m2023-07-24[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m│ [0;0m032-96684 [0;0m │ [0;0mmacOS Big Sur [0;0m │ [0;0m11.7.8 [0;0m │ [0;0m20G1351[0;0m │ [0;0m12.42 GB[0;0m │ [0;0m2023-06-28[0;0m │ [0;0mTrue      [0;0m │
[0;0m[0;0m└────────────┴────────────────┴─────────┴─────────┴──────────┴────────────┴────────────┘
[0;0m
2023-11-22 15:46:23 | v31.0 | [run_mist] Checking that selected version 13.6.2 is available
2023-11-22 15:46:23 | v31.0 | [run_mist] Setting mist to only list compatible installers
2023-11-22 15:46:23 | v31.0 | [get_mist_list] Non-default catalog selected (darwin version 22)

2023-11-22 15:46:23 | v31.0 | [run_mist] This command is now being run:

2023-11-22 15:46:23 | v31.0 | mist download installer 13.6.2 application --application-name Install %NAME%.app --output-directory /Applications --compatible --no-ansi --catalog-url https://swscan.apple.com/content/catalogs/others/index-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
┌──────────────────┐
│ INPUT VALIDATION │
└──────────────────┘
  ├─ User is 'root'...
  ├─ Download search string will be '13.6.2'...
  ├─ Include betas in search results will be 'false'...
  ├─ Only include compatible installers will be 'true'...
  ├─ Cache downloads will be 'false'...
  ├─ Output directory will be '/Applications'...
  ├─ Temporary directory will be '/private/tmp/com.ninxsoft.mist'...
  ├─ Force flag has not been set, existing files will not be overwritten...
  ├─ Application name will be 'Install %NAME%.app'...
┌────────┐
│ SEARCH │
└────────┘
  ├─ Searching for macOS download '13.6.2'...
  └─ No macOS Installer found with '13.6.2', exiting...
2023-11-22 15:46:23 | v31.0 | [run_mist] No installer found. I guess nothing got downloaded.

2023-11-22 15:46:23 | v31.0 | [erase-install] attempting to terminate the 'caffeinate' process - Termination message indicates success

2023-11-22 15:46:23 | v31.0 | [finish] sending quit message to dialog (/var/tmp/dialog.mjV)
2023-11-22 15:46:24 | v31.0 | [finish] Script exit code: 1
Load failed: 5: Input/output error
Try running `launchctl bootstrap` as root for richer errors.
Nudge launch agent has been loaded. Your not getting away from me that easily
[STEP 4 of 6]
[STEP 5 of 6]
Inventory will be updated when all queued actions in Self Service are complete.
[STEP 6 of 6]
Unmounting file server...
grahampugh commented 9 months ago

@kMikaZu This Issue relates to the --ffi flag, but you didn't supply that flag.

You're trying to get 13.6.2 but that update is only available for a small subset of Mac. You would be better to use --os=13 to get the latest version available for that Mac.

kMikaZu commented 9 months ago

I guess the only good option then whether to see an update is available or not, is to check on a test-device in the Software Update pane? We want to keep control and update version per version, on a regular base (with testing).

grahampugh commented 9 months ago

If you want to deploy specific versions then you'll need to keep on top of what versions are current. I would advise to run erase-install.sh --list on a test device of the same Model to see what the latest compatible update available is. Checking softwareupdate is not quite as good, because erase-install only handles full installers, whereas softwareupdate may be offering delta updates.

grahampugh commented 9 months ago

As this issue has been hijacked by various different and unrelated potential issues and questions, it's no longer possible to track properly. I'll close it, and any of you still having problems with their own issues, please open a new issue - don't answer here.