Unpack DMG in pure mode

[YorikSar]

Current unpackdmg.sh is impure and will not work with sandbox. Consider using undmg that installs unpack hook.

[jcszymansk]

I started with undmg, but it doesn't work with APFS images. I tried 7zip but it doesn't work either. :(

[jcszymansk]

From what I see, the only other thing that possibly could work is apfs-fuse; but there would need to be a way to check whether the image is HFS in which case undmg could be used, or APFS.

And I don't know whether it would work in a sandbox either...

[YorikSar]

No, I wouldn’t expect anything that mints something to work in sandbox. I guess the only way is to implement this in undmg or find a similar tool that supports APFS already. From the looks of it, one should be able to use linux-apfs code for that…

I don’t think there’s much can be done here, in this repo. Maybe try undmg and fallback to hdiutil if not in sandbox?..

[jcszymansk]

I noticed that the Nixpkgs (p)7zip version is very outdated, current version seems to work with APFS images. It has some problems with symlinks though, so a little patching might be needed, but I hope I can manage it.

[osalbahr]

How does this compare against how Homebrew itself handles *.dmg files?

[jcszymansk]

Homebrew uses hdiutil, but they do not use sandbox or have a pure mode, so it's not an issue for them.

[jcszymansk]

Oh well, 7zip just won't compile with the Libsystem version available from Nixpkgs (the problem is with sys/stat.h which lacks the definition of utimensat).

So it seems that at least until another tool that could unpack APFS .dmg is found, or Libsystem in Nixpkgs updated hdiutil must remain...

[osalbahr]

Oh well, 7zip just won't compile with the Libsystem version available from Nixpkgs (the problem is with sys/stat.h which lacks the definition of utimensat).

So it seems that at least until another tool that could unpack APFS .dmg is found, or Libsystem in Nixpkgs updated hdiutil must remain...

This seems like a bug to me. Maybe the fix needs to be upstreamed to nixpkgs?

[osalbahr]

Current unpackdmg.sh is impure and will not work with sandbox. Consider using undmg that installs unpack hook.

Not sure that’s feasible, at least not fully (for now?):

error: only HFS file systems are supported

• matthewbauer/undmg#4

[jcszymansk]

This seems like a bug to me. Maybe the fix needs to be upstreamed to nixpkgs?

It looks like they need simply update, the Nixpkgs version is taken from darling-xnu and the current version already has utimensat. Obviously there may be further errors lurking ahead...

I'll try to notify the Nixpkgs' Libsystem maintainers, but not before Monday.

[osalbahr]

Yeah I think it’s a good idea to open an issue and @ the maintainers, whenever you have the time for it.

[jcszymansk]

After some more investigation I was able to use 7zip and it seems that it is able to unpack calibre (an APFS dmg) just fine, it also builds, so I pushed this, but it needs way more testing than I am able to do on a mobile link now...

EDIT: also, now it's x86 only, I can probably add nix code for arm64, but I don't have an arm64 mac, so I will be unable to test either...

EDIT2: reverted this, now dmg unpacking with 7zip is available on the feature/7zip-for-dmgs branch, soon I'll push there also arm64 support, but I can't test it at all.

EDIT3: now 7zip should support also arm64.

[YorikSar]

@jacekszymanski Thanks! It doesn't seem to work on arm though:

 % nix build --no-link --print-out-paths github:jacekszymanski/nixcasks/feature/7zip-for-dmgs#calibre
error: builder for '/nix/store/f0pywlma49cmdnhfddh803dsg7api1a9-7zip-23.01.drv' failed with exit code 2;
       last 10 log lines:
       > clang -arch arm64     -O2 -c -Werror -Wall -Wextra -Weverything -Wfatal-errors -Wno-poison-system-directories -DNDEBUG -D_REENTRANT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fPIC   -o b/m_arm64/Lzma2Dec.o ../../../../C/Lzma2Dec.c
       > clang -arch arm64     -O2 -c -Werror -Wall -Wextra -Weverything -Wfatal-errors -Wno-poison-system-directories -DNDEBUG -D_REENTRANT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fPIC   -o b/m_arm64/Lzma2DecMt.o ../../../../C/Lzma2DecMt.c
       > clang -arch arm64     -O2 -c -Werror -Wall -Wextra -Weverything -Wfatal-errors -Wno-poison-system-directories -DNDEBUG -D_REENTRANT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fPIC   -o b/m_arm64/Lzma2Enc.o ../../../../C/Lzma2Enc.c
       > clang -arch arm64     -O2 -c -Werror -Wall -Wextra -Weverything -Wfatal-errors -Wno-poison-system-directories -DNDEBUG -D_REENTRANT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fPIC   -o b/m_arm64/LzmaDec.o -DZ7_LZMA_DEC_OPT ../../../../C/LzmaDec.c
       > clang -arch arm64     -O2 -c -Werror -Wall -Wextra -Weverything -Wfatal-errors -Wno-poison-system-directories -DNDEBUG -D_REENTRANT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fPIC   -o b/m_arm64/LzmaEnc.o ../../../../C/LzmaEnc.c
       > ../../../../C/LzmaEnc.c:198:21: fatal error: empty expression statement has no effect; remove unnecessary ';' to silence this warning [-Wextra-semi-stmt]
       >   BSR2_RET(pos, res);
       >                     ^
       > 1 error generated.
       > make: *** [../../7zip_gcc.mak:1143: b/m_arm64/LzmaEnc.o] Error 1
       For full logs, run 'nix-store -l /nix/store/f0pywlma49cmdnhfddh803dsg7api1a9-7zip-23.01.drv'.
error: 1 dependencies of derivation '/nix/store/jj95x7pffiviziayazkg2apdwcgjp6i2-calibre-6.24.0.drv' failed to build

[jcszymansk]

I had a similar error on x86 and was too lazy to go through their makefiles, so I only patched that single semicolon 🤷 and hoped it will be enough for arm as well.

Now I added a flag to make, so it should ignore this specific error (a warning indeed, but they have -Werror)

[jcszymansk]

I have pushed the updated feature branch, now it should work for all DMGs using 7zip. At least it works for those some-teen which I checked. But all the checking was on an Intel mac.

[dfrankland]

I'm testing on an arm mac and I could get some applications, like .#authy, to work, but others like .#docker did not

[jcszymansk]

Hi,

this issue is only about DMG unpacking. Docker will not work now on arm (and older x86 macos versions) as it uses variant binaries which is not yet supported.

[dfrankland]

Thanks for responding. I'm specifically talking about the DMG for Docker—7zip will not unpack it.

[jcszymansk]

OK, I see. Indeed 7zip won't unpack Docker's dmg and neither will undmg (which is unsuitable anyway).

7zz l Docker.dmg returns this crazy thing:

7-Zip (z) 23.01 (x64) : Copyright (c) 1999-2023 Igor Pavlov : 2023-06-20
 64-bit locale=en_US.UTF-8 Threads:4 OPEN_MAX:256

Scanning the drive for archives:
1 file, 656801294 bytes (627 MiB)

Listing archive: Docker.dmg

--
Path = Docker.dmg
Type = Dmg
Physical Size = 656801294
Method = Zero0 Zero2 2147483656 CRC
Blocks = 2143
Comment = Docker

   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
                    .....          512           84  0.MBR
                    .....          512          132  1.Primary GPT Header
                    .....        16384          248  2.Primary GPT Table
                    .....         3072            0  3.free
                    .....    209715200          884  4.efi_sys
                    .....   2172608512    655684592  5.hfs
                    .....    134221312            0  6.free
                    .....        16384          248  7.Backup GPT Table
                    .....          512          132  8.Backup GPT Header
------------------- ----- ------------ ------------  ------------------------
                            2516582400    655686320  9 files

Now I think I'll post an issue to 7zip about this, but until they fix it not much can be done here.

I'll write a fallback, so if 7zip fails, hdiutil will be used. Obviously applications which need this hack won't build in the sandbox.

[YorikSar]

I'll write a fallback, so if 7zip fails, hdiutil will be used. Obviously applications which need this hack won't build in the sandbox.

It would be great if you could mark such derivations with __noChroot = true so that Nix would throw an error if sandbox is set to true in config or build without sandbox if it's set to relaxed.

[jcszymansk]

I'm working (not on this branch, and I haven't pushed it yet) on an overrides mechanism, so it will be possible to apply specific attributes to individual derivations.

[jcszymansk]

FWIW, I posted a bug on 7zip: https://sourceforge.net/p/sevenzip/bugs/2411/

[jcszymansk]

It's strange, but hdiutil wouldn't attach the Docker dmg when called from nix, I don't know why. So for now, if an image fails to test with 7zip, I'm converting it to UDBZ (bzip2 compressed) and unpacking with 7zip, it seems to work this way, also it doesn't require mounting anything (but it still breaks the sandbox, as hdiutil is needed to convert the image).

[jcszymansk]

The next version of 7zip will support ULMO images, such as the Docker one.

As for this branch, I think I'll merge it and move on; when the next 7zip is released, I will remove the hdiutil hack.

[jcszymansk]

The branch has been merged, I'm closing this issue.

The next 7zip release is probably several months away.