search

munki / macadmin-scripts

Scripts of possible interest to macOS admins
Other
2.33k stars 523 forks source link

Getting Invalid XML & 'UNKNOWN' build numbers #66

Closed apizz closed 4 years ago

apizz commented 4 years ago

Tried when current working directly as ~ and /Users/Shared/ but in both cases produced the following output.

This is on macOS 10.15.6 (19G2021)

Mac:~ admin$ sudo installinstallmacos.py --compress
Downloading https://swscan.apple.com/content/catalogs/others/index-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog...
Downloading http://swcdn.apple.com/content/downloads/59/18/001-15219-A_EE9FN7UHJA/efuz3h0zlmx4qlinf4bxlguigo9jvhcr55/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/59/18/001-15219-A_EE9FN7UHJA/efuz3h0zlmx4qlinf4bxlguigo9jvhcr55/001-15219.English.dist...
Invalid XML in ./content/downloads/59/18/001-15219-A_EE9FN7UHJA/efuz3h0zlmx4qlinf4bxlguigo9jvhcr55/001-15219.English.dist
Downloading http://swcdn.apple.com/content/downloads/41/56/061-94403-A_5AM6QB8TGK/ijdhki76d4rw19t579eblaqc26ngzlhu1p/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/41/56/061-94403-A_5AM6QB8TGK/ijdhki76d4rw19t579eblaqc26ngzlhu1p/061-94403.English.dist...
Invalid XML in ./content/downloads/41/56/061-94403-A_5AM6QB8TGK/ijdhki76d4rw19t579eblaqc26ngzlhu1p/061-94403.English.dist
Downloading http://swcdn.apple.com/content/downloads/34/54/041-88800-A_HLMBDM42FL/anrmoj880qkj0lbybqm0c3830p70nawjrv/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/34/54/041-88800-A_HLMBDM42FL/anrmoj880qkj0lbybqm0c3830p70nawjrv/041-88800.English.dist...
Invalid XML in ./content/downloads/34/54/041-88800-A_HLMBDM42FL/anrmoj880qkj0lbybqm0c3830p70nawjrv/041-88800.English.dist
Downloading http://swcdn.apple.com/content/downloads/41/23/061-26578-A_FF0C6M87LS/ylvl0phx6uy3i444qt9p57n8wz0xnylmd9/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/41/23/061-26578-A_FF0C6M87LS/ylvl0phx6uy3i444qt9p57n8wz0xnylmd9/061-26578.English.dist...
Invalid XML in ./content/downloads/41/23/061-26578-A_FF0C6M87LS/ylvl0phx6uy3i444qt9p57n8wz0xnylmd9/061-26578.English.dist
Downloading http://swcdn.apple.com/content/downloads/23/45/001-36735-A_IRCLJ5YYKF/z8vka6xfykydycui5b7atpc52btxmbffdn/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/23/45/001-36735-A_IRCLJ5YYKF/z8vka6xfykydycui5b7atpc52btxmbffdn/001-36735.English.dist...
Invalid XML in ./content/downloads/23/45/001-36735-A_IRCLJ5YYKF/z8vka6xfykydycui5b7atpc52btxmbffdn/001-36735.English.dist
Downloading http://swcdn.apple.com/content/downloads/08/44/041-90855-A_452WLVQACP/mzfvoi59bljh3w4w4lvwenfk0r1a2f7212/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/08/44/041-90855-A_452WLVQACP/mzfvoi59bljh3w4w4lvwenfk0r1a2f7212/041-90855.English.dist...
Invalid XML in ./content/downloads/08/44/041-90855-A_452WLVQACP/mzfvoi59bljh3w4w4lvwenfk0r1a2f7212/041-90855.English.dist
Downloading http://swcdn.apple.com/content/downloads/06/50/041-91758-A_M8T44LH2AW/b5r4og05fhbgatve4agwy4kgkzv07mdid9/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/06/50/041-91758-A_M8T44LH2AW/b5r4og05fhbgatve4agwy4kgkzv07mdid9/041-91758.English.dist...
Invalid XML in ./content/downloads/06/50/041-91758-A_M8T44LH2AW/b5r4og05fhbgatve4agwy4kgkzv07mdid9/041-91758.English.dist
Downloading http://swcdn.apple.com/content/downloads/01/28/061-86291-A_JPEIWIOZES/enpozvvbmj3mj2dhulhevlt8b429qd5kw0/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/01/28/061-86291-A_JPEIWIOZES/enpozvvbmj3mj2dhulhevlt8b429qd5kw0/061-86291.English.dist...
Invalid XML in ./content/downloads/01/28/061-86291-A_JPEIWIOZES/enpozvvbmj3mj2dhulhevlt8b429qd5kw0/061-86291.English.dist
Downloading http://swcdn.apple.com/content/downloads/10/06/001-36801-A_9MBEQ97Z9M/672a1q4u8978ize3s15ygivp2dqwmk10qb/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/10/06/001-36801-A_9MBEQ97Z9M/672a1q4u8978ize3s15ygivp2dqwmk10qb/001-36801.English.dist...
Invalid XML in ./content/downloads/10/06/001-36801-A_9MBEQ97Z9M/672a1q4u8978ize3s15ygivp2dqwmk10qb/001-36801.English.dist
Downloading http://swcdn.apple.com/content/downloads/17/32/061-26589-A_8GJTCGY9PC/25fhcu905eta7wau7aoafu8rvdm7k1j4el/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/17/32/061-26589-A_8GJTCGY9PC/25fhcu905eta7wau7aoafu8rvdm7k1j4el/061-26589.English.dist...
Invalid XML in ./content/downloads/17/32/061-26589-A_8GJTCGY9PC/25fhcu905eta7wau7aoafu8rvdm7k1j4el/061-26589.English.dist
Downloading http://swcdn.apple.com/content/downloads/37/41/001-04366/a6boefh8b6bee4k2x6v5lwyfjr7od2ica3/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/37/41/001-04366/a6boefh8b6bee4k2x6v5lwyfjr7od2ica3/001-04366.English.dist...
Invalid XML in ./content/downloads/37/41/001-04366/a6boefh8b6bee4k2x6v5lwyfjr7od2ica3/001-04366.English.dist
 #      ProductID    Version    Build   Post Date  Title
 1      001-15219    10.15.5  UNKNOWN  2020-06-15  macOS Catalina
 2      001-04366    10.15.4  UNKNOWN  2020-05-04  macOS Catalina
 3      041-91758    10.13.6  UNKNOWN  2019-10-19  macOS High Sierra
 4      041-88800    10.14.4  UNKNOWN  2019-10-23  macOS Mojave
 5      061-26589    10.14.6  UNKNOWN  2019-10-14  macOS Mojave
 6      001-36735    10.15.6  UNKNOWN  2020-08-06  macOS Catalina
 7      061-94403    10.15.6  UNKNOWN  2020-07-15  macOS Catalina
 8      061-86291    10.15.3  UNKNOWN  2020-03-23  macOS Catalina
 9      041-90855    10.13.5  UNKNOWN  2019-10-23  Install macOS High Sierra Beta
10      061-26578    10.14.5  UNKNOWN  2019-10-14  macOS Mojave
11      001-36801    10.15.6  UNKNOWN  2020-08-12  macOS Catalina
gregneagle commented 4 years ago

Sounds like Apple has some invalid XML! Not much I/we can do about that (except report it: good luck and god bless should you decide to do that).

I'm seeing only one bad file currently: "Invalid XML in ./content/downloads/10/06/001-36801-A_9MBEQ97Z9M/672a1q4u8978ize3s15ygivp2dqwmk10qb/001-36801.English.dist"

Downloading https://swscan.apple.com/content/catalogs/others/index-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog...
Downloading http://swcdn.apple.com/content/downloads/59/18/001-15219-A_EE9FN7UHJA/efuz3h0zlmx4qlinf4bxlguigo9jvhcr55/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/59/18/001-15219-A_EE9FN7UHJA/efuz3h0zlmx4qlinf4bxlguigo9jvhcr55/001-15219.English.dist...
Downloading http://swcdn.apple.com/content/downloads/41/56/061-94403-A_5AM6QB8TGK/ijdhki76d4rw19t579eblaqc26ngzlhu1p/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/41/56/061-94403-A_5AM6QB8TGK/ijdhki76d4rw19t579eblaqc26ngzlhu1p/061-94403.English.dist...
Downloading http://swcdn.apple.com/content/downloads/34/54/041-88800-A_HLMBDM42FL/anrmoj880qkj0lbybqm0c3830p70nawjrv/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/34/54/041-88800-A_HLMBDM42FL/anrmoj880qkj0lbybqm0c3830p70nawjrv/041-88800.English.dist...
Downloading http://swcdn.apple.com/content/downloads/41/23/061-26578-A_FF0C6M87LS/ylvl0phx6uy3i444qt9p57n8wz0xnylmd9/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/41/23/061-26578-A_FF0C6M87LS/ylvl0phx6uy3i444qt9p57n8wz0xnylmd9/061-26578.English.dist...
Downloading http://swcdn.apple.com/content/downloads/23/45/001-36735-A_IRCLJ5YYKF/z8vka6xfykydycui5b7atpc52btxmbffdn/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/23/45/001-36735-A_IRCLJ5YYKF/z8vka6xfykydycui5b7atpc52btxmbffdn/001-36735.English.dist...
Downloading http://swcdn.apple.com/content/downloads/08/44/041-90855-A_452WLVQACP/mzfvoi59bljh3w4w4lvwenfk0r1a2f7212/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/08/44/041-90855-A_452WLVQACP/mzfvoi59bljh3w4w4lvwenfk0r1a2f7212/041-90855.English.dist...
Downloading http://swcdn.apple.com/content/downloads/06/50/041-91758-A_M8T44LH2AW/b5r4og05fhbgatve4agwy4kgkzv07mdid9/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/06/50/041-91758-A_M8T44LH2AW/b5r4og05fhbgatve4agwy4kgkzv07mdid9/041-91758.English.dist...
Downloading http://swcdn.apple.com/content/downloads/01/28/061-86291-A_JPEIWIOZES/enpozvvbmj3mj2dhulhevlt8b429qd5kw0/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/01/28/061-86291-A_JPEIWIOZES/enpozvvbmj3mj2dhulhevlt8b429qd5kw0/061-86291.English.dist...
Downloading http://swcdn.apple.com/content/downloads/10/06/001-36801-A_9MBEQ97Z9M/672a1q4u8978ize3s15ygivp2dqwmk10qb/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/10/06/001-36801-A_9MBEQ97Z9M/672a1q4u8978ize3s15ygivp2dqwmk10qb/001-36801.English.dist...
Invalid XML in ./content/downloads/10/06/001-36801-A_9MBEQ97Z9M/672a1q4u8978ize3s15ygivp2dqwmk10qb/001-36801.English.dist
Downloading http://swcdn.apple.com/content/downloads/17/32/061-26589-A_8GJTCGY9PC/25fhcu905eta7wau7aoafu8rvdm7k1j4el/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/17/32/061-26589-A_8GJTCGY9PC/25fhcu905eta7wau7aoafu8rvdm7k1j4el/061-26589.English.dist...
Downloading http://swcdn.apple.com/content/downloads/37/41/001-04366/a6boefh8b6bee4k2x6v5lwyfjr7od2ica3/InstallAssistantAuto.smd...
Downloading https://swdist.apple.com/content/downloads/37/41/001-04366/a6boefh8b6bee4k2x6v5lwyfjr7od2ica3/001-04366.English.dist...
 #      ProductID    Version    Build   Post Date  Title
 1      001-15219    10.15.5  19F2200  2020-06-15  macOS Catalina
 2      001-04366    10.15.4  19E2269  2020-05-04  macOS Catalina
 3      041-91758    10.13.6    17G66  2019-10-19  macOS High Sierra
 4      041-88800    10.14.4  18E2034  2019-10-23  macOS Mojave
 5      061-26589    10.14.6   18G103  2019-10-14  macOS Mojave
 6      001-36735    10.15.6  19G2006  2020-08-06  macOS Catalina
 7      061-94403    10.15.6    19G73  2020-07-15  macOS Catalina
 8      061-86291    10.15.3  19D2064  2020-03-23  macOS Catalina
 9      041-90855    10.13.5   17F66a  2019-10-23  Install macOS High Sierra Beta
10      061-26578    10.14.5  18F2059  2019-10-14  macOS Mojave
11      001-36801    10.15.6  UNKNOWN  2020-08-12  macOS Catalina

Choose a product to download (1-11):
hkatsura commented 4 years ago

.dist is now compressed (gz), it wasn't compressed before, i assume?

$ curl --head https://swdist.apple.com/content/downloads/49/42/001-43966-A_UZVPNAGC5P/smuubcmj04qay4n78q85cpe85rcsdu00qy/001-43966.English.dist HTTP/2 200 server: Apache last-modified: Tue, 01 Sep 2020 00:12:39 GMT etag: "205e-5ae3560845c1b" accept-ranges: bytes content-type: application/vnd.apple.installer+xml content-language: en-us vary: Accept-Encoding content-encoding: gzip cache-control: max-age=60 expires: Wed, 23 Sep 2020 21:50:58 GMT date: Wed, 23 Sep 2020 21:49:58 GMT content-length: 2747

$ curl -s -O https://swdist.apple.com/content/downloads/49/42/001-43966-A_UZVPNAGC5P/smuubcmj04qay4n78q85cpe85rcsdu00qy/001-43966.English.dist $ file 001-43966.English.dist 001-43966.English.dist: gzip compressed data, from FAT filesystem (MS-DOS, OS/2, NT), original size modulo 2^32 8286 $ mv 001-43966.English.dist 001-43966.English.dist.gz $ gunzip 001-43966.English.dist.gz $ head 001-43966.English.dist 

<?xml version="1.0" encoding="utf-8"?>
<installer-gui-script minSpecVersion="2">
    <pkg-ref id="InstallAssistant" packageIdentifier="com.apple.pkg.InstallAssistant.Seed.macOS1016Seed1"/>
    <pkg-ref id="MajorOSInfo" packageIdentifier="com.apple.pkg.MajorOSInfo"/>
    <pkg-ref id="Info.plist" packageIdentifier="com.apple.plist.Info"/>
    <pkg-ref id="InstallInfo.plist" packageIdentifier="com.apple.plist.InstallInfo"/>
    <pkg-ref id="UpdateBrain.zip" packageIdentifier="com.apple.zip.UpdateBrain"/>
    <pkg-ref id="BuildManifest.plist" packageIdentifier="com.apple.plist.BuildManifest"/>
    <title>macOS Big Sur Beta</title>
    <product-generator name="IASUProductGenerator" version="16"/>
gregneagle commented 4 years ago

PRs to handle this accepted! Though I suspect it would be better to specify the expected content-type in the request headers

gregneagle commented 4 years ago

BTW, this is arguably an Apple bug: they should not be sending a compressed response if we don't send a Accept-Encoding: gzip header in the request.

hkatsura commented 4 years ago

a quick (and lazy) fix (until a correct fix becomes available):

    ...
    curl_cmd = ['/usr/bin/curl', options, '--create-dirs',
                '-o', local_file_path]
    if full_url.endswith('.dist'):
        curl_cmd.append('--compressed')
    if not ignore_cache and os.path.exists(local_file_path):
    ...
gregneagle commented 4 years ago

But that fails if Apple returns a non-compressed file. Too many assumptions there for me to consider that a proper fix.

gregneagle commented 4 years ago

https://github.com/munki/macadmin-scripts/commit/13fb28cdc8e59aca2bcbebeb61590cefad9c04bc addresses this but the real issue is Apple is arbitrarily violating HTTP standards (which I suspect they feel is OK because their own client is the only one they actually support). But this means this could easily break again at any time.

gregneagle commented 4 years ago

Oh -- you'll have to remove any affected files from the local cache and let the tool re-download them.