Manage Perl modules with "Universal Binary" (e.g. arm64, x86_64)

[systemresearch]

Does App::perlbrew support the management of the *.bundle macOS "Universal Binary" (e.g. arm64, x86_64) files in Perl modules?

Related:

• "StackOverflow: Install & update a Perl module as "universal" (x86_64, arm64)?"
• ExtUtils::MakeMaker -- Generate a Makefile with "Universal Binary" (e.g. arm64, x86_64) directives? #445

[gugod]

perlbrew itself does not do compilation of Perl modules but only tweaks enough env variables to make cpanm or cpan install modules to the right place -- either a "site_perl" directory or a local::lib directory.

Also the complication setup of a module depends on the content of Makerile.PL or Build.PL of the module. To my knowledge, I don't think any comonnly-used, simple, setup would produce universal binaries at the end.

The "ARCHFLAG" approach you've mentioned in https://github.com/Perl-Toolchain-Gang/ExtUtils-MakeMaker/issues/445 seems to lead to some viable solution. But currently this variable is not special to perlbrew.

[systemresearch]

I did find that ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' worked OK with cpanm. Also, on macOS, binary Perl .bundle modules can be found and checked with a find /Library/Perl -name "*.bundle" | xargs file approach.

So, while perlbrew does not internally have any specialized code for dealing with universal binaries, the questions to consider might be:

• Is perlbrew compatible with an env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' perlbrew … approach?
• And, if yes, should this particular use case be mentioned in the perlbrew docs?

As FYI, I am not currently a user of perlbrew. I found perlbrew in a broad survey for understanding and possible tools for dealing with the Perl univeral binary issues. Thus, I had posted the issue|questions here. Thank you for your reply.

[gugod]

• Is perlbrew compatible with an env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' perlbrew … approach?

I haven't test it myself but I believe so. Since ARCHFLAGS is not specially handled by perlbrew itself, by properly export-ing this variable in the shell, the compilation process triggered by perlbrew install should be able to see this variable and respond correctly.

• And, if yes, should this particular use case be mentioned in the perlbrew docs?

Since perlbrew itself is meant for personal use on a single computer instead of producing / distributing binary builds, I see little value of building universal-binaries. But maybe that's something people would want. It would still be nice to mention this approach (once verified) in a document. Eventually we will find it to be useful. :)

[Leont]

Is perlbrew compatible with an env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' perlbrew … approach?

Why use ARCHFLAGS and not put it in ccflags/ldflags/lddlflags?

[systemresearch]

@gugod Good point on scope, here, there situation...

... personal use on a single computer instead of producing / distributing binary builds ...

For clarification, this issue actually is a "personal use on a single computer" situation. I'm not trying to create any binaries for distribution.

Goal: personal use, single user, standard-release Perl installation without Instruction Set Architecture (ISA) runtime issues... on Rosetta enabled macOS.

After a user has enabled Rosetta 2 on an Apple Silicon computer, there are two Instruction Set Architectures (ISA) which automatically dispatch without requiring any additional user interaction.

A Rosetta 2 enabled Apple Silicon computer has automatic dispatch on either the native arm64 (arm64e) ISA or the rosetta x86_64 ISA from the mach-o *.bundle binary. Any "Universal" mach-o *.bundle binaries which include arm64/arm64e/x86_64 will avoid ISA runtime issues for the general (technical and non-technical) user.

At a technical level, which binaries slices (arm64|arm64e|x86_64) are in each *.bundle can be found from the macOS terminal command line:

BREW_INSTALL_DIR = /Library/Perl
find $BREW_INSTALL_DIR -name "*.bundle" | xargs file
# -- or, check the whole system --
find / -name "*.bundle" 2> /dev/null | xargs file

[systemresearch]

@Leont A shapshot of findings relative to the current state of "universal binaries" and perl...

Why use ARCHFLAGS and not put it in ccflags/ldflags/lddlflags?

Good question… there appears to be a least three approaches to create mach-o universal binary loadable library bundles in the Perl eco-system:

• ARCHFLAGS
• lipo - universal binary tool
• ccflags/ldflags/lddlflags

ARCHFLAGS

Use of ARCHFLAGS was (re-)discovered in the current macOS 13.3 Ventura (but long-ago written) perlmacosx man page.

The above information is clearly not recent because it speaks to PowerPC/Intel and 32/64 bit support. Even so, the above information was found to still successfully work for Perl as follows:

env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' perl Makefile.PL

make
### An Apple Silicon machine can test all three architectures:
arch -x86_64 make test
arch -arm64  make test
arch -arm64e make test
### An Apple Intel machine can only test its native architecture:
make test

(sudo) make install

An addition, I later found that ARCHFLAGS worked with a cpanm install of Finance::Quote for GnuCash.

(sudo) env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpan App::cpanminus
(sudo) env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpanm Test2
(sudo) env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpanm Finance::Quote
(sudo) env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpanm JSON::Parse

### verify universal binaries
find /Library/Perl -name "*.bundle" | xargs file

_Side note: It was @Leont comment "If you can make -arch x86_64 -arch arm64 work…" that lead to my grepping terms like arch x86_64 which found the forgotten & obscure perlmacosx man page._

lipo - universal binary tool

The current Apple Developer documentation for "Building a Universal macOS Binary" provides information to use the lipo tool in the makefile:

x86_app: main.c
    $(CC) main.c -o x86_app -target x86_64-apple-macos10.12
arm_app: main.c
    $(CC) main.c -o arm_app -target arm64-apple-macos11
universal_app: x86_app arm_app
    lipo -create -output universal_app x86_app arm_app

Note: You can build a universal binary on either an Apple silicon or Intel-based Mac computer, but you cannot debug the arm64 slice of your binary on an Intel-based Mac computer. It’s possible to debug both slices of a universal binary on Apple silicon.

man lipo

ccflags/ldflags/lddlflags

-- Proof of Concept --

A proof-of-concept ccflags/ldflags/lddlflags example did demonstrate that Perl [arm64, x86_64] univeral binaries could be created. For me, this indicated that Perl universal binaries are possible … that was great news! However, a hand-edit-the-build-sequence approach did not look convenient to use on a general wide scale.

#! /bin/bash
arch_opt="-arch x86_64 -arch arm64"
opt1="-I./Encode -fno-common -DPERL_DARWIN -fno-strict-aliasing"
opt2="-mmacosx-version-min=12.0 -fstack-protector-strong"
opt3="-pipe -DPERL_USE_SAFE_PUTENV -Wno-error=implicit-function-declaration -O3"
opt4=-DVERSION=\"3.19\"
opt5=-DXS_VERSION=\"3.19\"
opt6="-I/opt/homebrew/Cellar/perl/5.34.0/lib/perl5/5.34.0/darwin-thread-multi-2level/CORE"
ccopts="$arch_opt $opt1 $opt2 $opt3 $opt4 $opt5 $opt6"
ldopts="$arch_opt $opt2"
cc -c  $ccopts Encode.c
cc -c $ccopts def_t.c
cc -c $ccopts encengine.c
cc -bundle -undefined dynamic_lookup $ldopts \
   Encode.o def_t.o encengine.o -o blib/arch/auto/Encode/Encode.bundle
file blib/arch/auto/Encode/Encode.bundle

_-- ExtUtils::FakeConfig Configu --

The next finding was ExtUtils::FakeConfig Config_u.pm which also used ccflags and lddlflags:

package Config_u;

require ExtUtils::FakeConfig;
require Config;

my %values =
  ( lddlflags => ' -arch i386 -arch ppc ' . $Config::Config{lddlflags},
    ccflags   => ' -arch i386 -arch ppc ' . $Config::Config{ccflags},
    );

ExtUtils::FakeConfig->import( %values );

The MConfig_u use synopsis looked simple enough:

perl -MConfig_u Makefile.PL
make
make test
make install

However, MConfig_U was only maintained from 2002 to 2008 .AND. had a caveat that lipo would be the "safest" approach:

Note that the safest way to build Universal binaries is to compile the modules separately and then use lipo(1) to merge the resulting .bundle files.

_Note: This was the point in time where I inquired if ExtUtils::MakeMaker could "Generate a Makefile with "Universal Binary" (e.g. arm64, x86_64) directives?". It seemed like the place where Makefile.pl transitions to Makefile would be generally helpful to the larger (macOS) Perl ecosystem._

-- Perl Variables --

It was news to me that ccflags/ldflags/lddlflags might be built into perl itself...

What does perl -V:ccflags -V:ldflags -V:lddlflags return?

Ideally, you would put them in each of those variables when building perl, then you don't have to worry about it later.

Interesting. Yet, puzzles remain:

• Could this really be a easy as adding the flags to perl for the general community?
  • Or, does Apple know some reason to only add the -bundle flag to perl?
• General releases of Apple's system perl and Homebrew perl had only a -bundle flag to indicate creation of a mach-o bundle. Otherwise, no universal binary flags are present.
• Creating and maintaining one's own build of perl seems like a significant undertaking.

Summary

1. ARCHFLAGS. I'm currently using the ARCHFLAGS approach because it works at a good-enough level for an end user of software which uses perl. Successful work-around status.
2. lipo. A makefile with lipo is the stated (preferred?) approach in Apple developer documents. However, this approach has not been found to be in use for perl.
3. ccflags/ldflags/lddlflags. This would be OK if the flag details became part of general perl release and worked with the downstream Makefile.PL/Makefile builds. … however, it's unclear how to get the right persons' attention on this option.

[systemresearch]

@Leont Switching from a user to implementer perspective...

Why use ARCHFLAGS and not put it in ccflags/ldflags/lddlflags?

Perhaps a meaningful difference is that ARCHFLAGS have broader designation within a given Makefile for flow & logic than ccflags/ldflags/lddlflags? i.e. ccflags/ldflags/lddlflags have a more narrow designation to the flags of a specific build tool.

Some GitHub related code searches:

• /(^|\s)ARCHFLAGS([^\w])/ org:apple
• /(^|\s)ARCHFLAGS([^\w])/ org:llvm
• /(^|\s)ARCHFLAGS([^\w])/ language:Makefile x86_64 arm64
• /(^|\s)ARCHFLAGS([^\w])/ language:Makefile lipo

[jlholt]

Is perlbrew compatible with an env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' perlbrew … approach?

Why use ARCHFLAGS and not put it in ccflags/ldflags/lddlflags?

Uniformity is a good thing. A uniform way of building perl and building perl modules is ideal.

For example, if I wanted to create a macOS distribution of perl 5.32.1 and then install modules that are also universal and then distribute that perl so that both x86_64 and arm64e customers could use it, then I'd want to use the same mechanism to perform all operations so that universal binaries were produced. I think that's the first thing people would think of: one way to identify my intentions for all compilations.