MacOS universal binaries?

[plk]

I created two pp binaries, one for x86_64 and one for ARM, signed them both, both work. Then combined them into a universal binary with lipo, resulting in new binary. This only works on ARM and the x86_64 fails with errors like this:

Compilation failed in require at /loader/HASH(0x7fdba0884a10)/XSLoader.pm line 118.

Just wanted to confirm that this won't work due to the assumptions pp'ed binaries make about headers etc. I assume that there is no way to create universal binaries with pp?

[rschupp]

Just wanted to confirm that this won't work due to the assumptions pp'ed binaries make about headers etc. I assume that there is no way to create universal binaries with pp?

Sorry, there's no way to create "universal" binaries. See the section "Anatomy of a Self-Contained PAR executable" in PAR::Tutorial: you would somehow have to combine the corresponding parts of two native packed executables separately (e.g. use lipo for the first (execuatble) parts, create a "union" zip from the third (zip) parts), then make sure that when run on architecture A, only the stuff for this architecture from the "union" zip is used etc.

I'm curious why our naive attempt works at all for at least one architecture. My guess is that lipo works by concatenating the two executables and then prepends a new MachO header that references sections in both executables (tagged with architecture) and the MacOS loader picks the correct sections. In this scenario the pp binary the was placed last in the resulting universal binary might actually work (since the "embedded" files are correctly found by offset from the end of the file, same goes for the zip).

[plk]

I think this is correct as in testing using lipo, it's the last one in the universal list that seems to work. It's not that it works on ARM vs x86_64, it's just that the ARM binary was after the x86_64. I wonder if there is a way then to detect universal binaries and adjust the unpacking offsets at runtime.

[rschupp]

I wonder if there is a way then to detect universal binaries and adjust the unpacking offsets at runtime.

There are two problems here:

• The zip file is extracted using Archive::Zip: it always searches backwards from the end of the file until it finds a certain signature, then locates the zip "directory" from there. AFAIK there's no way to direct Archive::Zip to extract a zip not in the tail position.
• The "embedded" files are found using information at a fixed offset from the end of the file. How can you find this information for the binary in the front position?

[rschupp]

AFAIK there's no way to direct Archive::Zip to extract a zip not in the tail position.

Correction: there might be a away, see https://metacpan.org/release/PHRED/Archive-Zip-1.68/source/examples/readScalar.pl

But then we're facing the same problem as for the "embedded" files.

[plk]

I wonder if we can then detect a fat binary, if it's a fat binary, use lipo (which should be on any MacOS I think?) to extract that arch and then run as normal? lipo can do this by extracting the arch (given by the arch command) to a new file - I did a quick test and this works - the pp binary runs as normal after such an extract.

[plk]

There is information in the FAT header about the offsets for the different archictectures:

[rschupp]

You could patch something like the following into myldr/boot.c, directly before the call of extract_embedded_file:

• open my_prog for reading
• read the MachO header
• if it indicates a fat binary with more than one architecture:
  • use lipo to extract a "lean" binary from my_prog, preferably into some file in stmpdir
  • execve the new file with the original argv[1:]
• otherwise proceed as usual

[plk]

Opened a PR doing what was suggested - works on my previously failing tests.

[rschupp]

Closed via #58