search

ninxsoft / mist-cli

A Mac command-line tool that automatically downloads macOS Firmwares / Installers.
MIT License
619 stars 29 forks source link

Allow for cache of downloaded packages #67

Closed n8felton closed 2 years ago

n8felton commented 2 years ago

While doing testing, or when deciding to generate multiple output packages (perhaps DMG first, and ISO next), it would be nice if there was an option to cache and/or not delete the files downloaded.

A couple of factors come to mind:

In looking at the sucatalog, we could verify the cache using the Size key. (I'm sure there is a way to do this in Swift without shelling out to stat, but as an example)

      <key>Size</key>
      <integer>12225260436</integer>
% stat -f '%Dz' InstallAssistant.pkg
12225260436

The Digest key is also available, but I have yet to determine how the Digest is calculated. My assumptions so far are that it's a SHA1 hash, and that it's for InstallAssistant.pkg (given the Size key matches the same file).

      <key>Digest</key>
      <string>0646ab8257b829e1ad9e25c373b3b39ae9c97289</string>

but I get

% openssl dgst -sha1 InstallAssistant.*                                                                                                    
SHA1(InstallAssistant.pkg)= e1fd7b104e6a6b333db4581a6b4eede9268bf5b1
SHA1(InstallAssistant.pkg.integrityDataV1)= 09365ce5348c3e7817e2110244099993bcc81128
SHA1(InstallAssistant.pkm)= dff9477b17f960705da3ea84ea6173f7d82cd627

Example snip from sucatalog

<key>002-79225</key>
<dict>
  <array>
    <dict>
      <key>Digest</key>
      <string>0646ab8257b829e1ad9e25c373b3b39ae9c97289</string>
      <key>Size</key>
      <integer>12225260436</integer>
      <key>IntegrityDataURL</key>
      <string>https://swcdn.apple.com/content/downloads/49/07/002-79225-A_68Z2X28AVV/jfmg28xxxarr9av7ry0iefvst7cyl0qdhb/InstallAssistant.pkg.integrityDataV1</string>
      <key>MetadataURL</key>
      <string>https://swdist.apple.com/content/downloads/49/07/002-79225-A_68Z2X28AVV/jfmg28xxxarr9av7ry0iefvst7cyl0qdhb/InstallAssistant.pkm</string>
      <key>URL</key>
      <string>https://swcdn.apple.com/content/downloads/49/07/002-79225-A_68Z2X28AVV/jfmg28xxxarr9av7ry0iefvst7cyl0qdhb/InstallAssistant.pkg</string>
      <key>IntegrityDataSize</key>
      <integer>42044</integer>
    </dict>
  </array>
</dict>

Let me know what you think.

ninxsoft commented 2 years ago

The ability to temporarily cache downloads will speed up my testing, which is probably reason enough to implement this feature!

I'm proposing the following:

Just need to figure out how the digest is calculated 🤔

ninxsoft commented 2 years ago

Confirming how the digest is calculated will also allow me to validate the digest, like I do for IPSW downloads:

https://github.com/ninxsoft/Mist/blob/ec462cff3ba4548ccdeddba46b465965cb8b3eb2/Mist/Helpers/Generator.swift#L46-L55

n8felton commented 2 years ago

This is pretty far into the weeds, and I'm not sure it helps, but I wanted to share some findings in case it leads someone else to discovering the secret sauce.

InstallAssistant.pkg.integrityDataV1 appears to be a "chunklist" file format.

% head -1 InstallAssistant.pkg.integrityDataV1
CNKL$?$??0?}}[?`???*y??D?!(E?8?+?

CNKL = 0x4C4B4E43 which = #define CHUNKLIST_MAGIC 0x4C4B4E43

https://opensource.apple.com/source/xnu/xnu-7195.81.3/bsd/kern/chunklist.h.auto.html https://opensource.apple.com/source/xnu/xnu-7195.81.3/bsd/kern/chunklist.c.auto.html

I also came across https://github.com/hack-different/go-aapl-integrity/blob/master/pkg/chunklist/chunklist.go.

Looking through the code on these 2 findings, and the file header on InstallAssistant.pkg.integrityDataV1, cl_sig_method is 2, which is defined in chunklist.go as const ChunklistSignatureMethodIntegrityData = 2, but is never used anywhere in the code.

Likewise, method 2 is not even defined in chunklist.h.

PicoMitchell commented 2 years ago

I also can't seem to figure out what format that Digest key is, or if it is SHA1 what file it's actually for.

But, thinking about simply validating an existing "InstallAssistant.pkg" file... along with making sure the file Size key from the sucatalog info matches the actual file size, you can check its signature with pkgutil --check-signature /path/to/InstallAssistant.pkg. It looks like it should have a certificate chain Software Update > Apple Software Update Certification Authority > Apple Root CA.

Since "InstallAssistant.pkg" files are both a package and a dmg (https://www.mothersruin.com/software/SuspiciousPackage/faq.html#package-dmg), you could also validate the dmg side of the file using its own stored checksums with hdiutil verify /path/to/InstallAssistant.pkg.

ninxsoft commented 2 years ago

I stumbled across this article from Apple: https://support.apple.com/en-us/guide/security/sec2512a0c09/web

The recoveryOS is completely separate from the main macOS and the entire contents are stored in a disk image file named BaseSystem.dmg. There is also an associated BaseSystem.chunklist, which is used to verify the integrity of the BaseSystem.dmg. The chunklist is a series of hashes for 10 MB chunks of the BaseSystem.dmg. The UEFI firmware evaluates the signature of the chunklist file and then evaluates the hash one chunk at a time from the BaseSystem.dmg. This helps ensure that it matches the signed content present in the chunklist. If any of these hashes don’t match, booting from the local recoveryOS is aborted and the UEFI firmware attempts to boot from Internet recoveryOS instead.

Might be something, might be nothing, but combined with the links @n8felton has provided I might fire up a hex editor and have a poke around...

ninxsoft commented 2 years ago

@n8felton thanks again for linking to go-aapl-integrity, I have been able to break down the contents of Info.plist.integrityDataV1:

Note: Values need to be converted to little endian.

Offset     Bytes         Description
------------------------------------
00 -> 03   43 4E 4B 4C   Magic Header (0x4C4B4E43)
04 -> 07   24 00 00 00   Total Header Size (0x24 / 36 bytes, 0x00 -> 0x23 is 0x24 / 36 bytes)
08 -> 0B   01 01 02 00   File Version (1) + Chunk Method (1) + Signature Method (2 = Integrity Data) + Padding
0C -> 0F   01 00 00 00   Total Chunk Count Start (1 x chunk total)
10 -> 13   00 00 00 00   Total Chunk Count End
14 -> 17   24 00 00 00   Chunk Offset Start (Chunks start at offset 0x24)
18 -> 1B   00 00 00 00   Chunk Offset End
1C -> 1F   48 00 00 00   Signature Offset Start (Signature starts at offset 0x48)
20 -> 23   00 00 00 00   Signature Offset End
----------------------   Array of Chunks Start
24 -> 27   3C 14 00 00   Chunk Size (0x143C / 5180 bytes, should match wc -c Info.plist)
28 -> 2B   D9 2C 08 0B   Chunk SHA256 Hash Start (should match shasum -a 256 Info.plist)
2C -> 2F   26 52 CA 9C        |
30 -> 33   14 E6 A1 B9        |
34 -> 37   C0 5B 85 E3        |
38 -> 3B   C3 6C 86 10        |
3C -> 3F   98 BC 5C 60        |
40 -> 43   10 0D 85 93        |
44 -> 47   1B 8A 64 05   Chunk SHA256 Hash End
----------------------   Array of Chunks End
48 -> 4B   5C 3C 4E 59   Signature Start
4C -> 4F   90 B5 7F B3        |
50 -> 53   99 86 EE E6        |
54 -> 57   CC 0A EB CB        |
58 -> 5B   2F 4C F3 C8        |
5C -> 5F   5A 94 15 8F        |
60 -> 63   30 38 38 1D        |
64 -> 67   C6 9E 05 99   Signature End

The same approach can be applied against files with more than 1 chunk (ie. InstallAssistant.pkg should have 0x048E / 1116 chunks according to InstallAssistant.pkg.integrityDataV1)

At a minimum this gives me a way to compare the hashes of downloaded files against the contents of the corresponding chunklists.

chunklist.go also has details on the signature, will need to investigate further to see what this needs to be compared against.

ninxsoft commented 2 years ago

Successfully reading in chunks from the downloaded files, and comparing generated shasums with the ones provided by the chunklists!

https://github.com/ninxsoft/Mist/tree/feature-cache-downloads

Caveat: The entire file (one chunk a a time) is being loaded into memory, which isn't ideal (~11GB for macOS Monterey). Need to investigate options to free memory during the validation sequence.

ninxsoft commented 2 years ago

Well that was easy, autoreleasepool to the rescue!

https://github.com/ninxsoft/Mist/blob/9da1c6f8535c8281b6f2e146f8a73f28660c1f1e/Mist/Helpers/Validator.swift#L33-L49

ninxsoft commented 2 years ago

Chunklists and/or integrity data files are represented slightly different for macOS Catalina (10.15) and macOS Mojave (10.14), and they seem to get more complicated for macOS High Sierra (10.13).

Considering we are so close to WWDC, I am opting to skip any custom logic for macOS High Sierra, and macOS Catalina + macOS Mojave get best efforts (compare file sizes and integrity data if found).