Error on Apple Silicon Mac - "configure: error: Unexpected output of 'arch' on OSX"

[paulmromer]

I am able to build with portable python on an Intel Mac, but not on an Apple Silicon Mac.

I'll post my experience below. Do you have any suggestions about things I should do to debug the problem?

Here is a summary of my experience:

[Environment - on the Apple Silicon iMac of a colleague (John Restrepo)] (venv3.10) johnrestrepo@Johns-iMac pp % printenv TMPDIR=/var/folders/ms/h5x3h0px009d_pbzpxg4w_wm0000gn/T/ __CFBundleIdentifier=com.apple.Terminal XPC_FLAGS=0x0 TERM=xterm-256color SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.UOQhWh2pix/Listeners XPC_SERVICE_NAME=0 TERM_PROGRAM=Apple_Terminal TERM_PROGRAM_VERSION=443 TERM_SESSION_ID=9964ACA4-FE1F-4651-B079-82CB4CB6EBC2 SHELL=/bin/zsh HOME=/Users/johnrestrepo LOGNAME=johnrestrepo USER=johnrestrepo PATH=/Users/johnrestrepo/coding/venv3.10/bin:/Library/Frameworks/Python.framework/Versions/3.10/bin:/Libary/Frameworks/Python.framework/Versions/3.10:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin SHLVL=1 PWD=/Users/johnrestrepo/coding/pp OLDPWD=/Users/johnrestrepo/coding VIRTUAL_ENV=/Users/johnrestrepo/coding/venv3.10 PS1=(venv3.10) %n@%m %1~ %# VIRTUAL_ENV_PROMPT=(venv3.10) LANG=enUS.UTF-8 =/usr/bin/printenv

[Command and error] (venv3.10) johnrestrepo@Johns-iMac pp %portable-python build 3.8.7 ... configure: error: Unexpected output of 'arch' on OSX ERROR Run failed: build/components/cpython/configure --prefix=/ppp-marker/3.8.7 --enable-optimizations --with-lto --with-ensurepip=upgrade --with-openssl=build/deps --with-system-ffi --with-dbmliborder=ndbm ./configure exited with code 1 INFO Compiling cpython took 10 seconds INFO Overall compilation took 53 seconds

[Confirmation of failure: No dist directory] (venv3.10) johnrestrepo@Johns-iMac pp % ls build

[zsimic]

Interesting, I am able to build on M1 without issue (did this partially to have an M1 build). I don't have my loaner M1 handy currently unfortunately, but I will over the week-end.

I'm thinking it's possibly caused by the fact that somehow portable-python ran ./configure here while in x86 mode somehow?

I will compare your output with mine, do send the contents of build/logs/00-portable-python.log if you have it, and perhaps also build/logs/*-cpython.log, that will help determine what may have caused this

[paulmromer]

I tried adding options to configure that specify the MacOS sdk: --enable-universalsdk --with-universal-archs=universal2

and ran: portable-python build 3.8.7

At the end, it gave the following error, which suggests that the current version of portable-python is not designed to work on the new M1 Macs because it does not offer an Arm-64 option:

configure: error: proper usage is --with-universal-arch=32-bit|64-bit|all|intel|3-way

ERROR Run failed: build/components/cpython/configure --prefix=/ppp-marker/3.8.7 --enable-optimizations --with-lto --with-ensurepip=upgrade --enable-universalsdk --with-universal-archs=universal2 --with-openssl=build/deps --with-system-ffi --with-dbmliborder=ndbm

./configure exited with code 1

[paulmromer]

Sorry, our messages crossed ...

I will get you the logs.

Is there anything else I should try as an option for configure?

[zsimic]

The idea is that it works well "out of the box", without you having to do anything :) Would love to know what was missed (or could perhaps be detected so a better error/explanation can be given in such a case, where for example parent process is running in x86 mode or something like that - I am not super familiar with M1 yet :) )

[zsimic]

Also, the output of portable-python diagnostics will be useful, especially the first part (it also shows a .yml config dump, which is probably irrelevant here in your case, since the failure is so early, doubt it's caused by any "config").

The first part of that diagnostics output will be very useful, especially the "invoker python" part (1st line)

[paulmromer]

I don't see a log folder under build. Do I need to run with some specific option to generate these?

I do see, for example, config.log under the /components/cpython directory

[zsimic]

Ah oops, forgot by default it doesn't record those logs. Can you uncomment this line if you're running from a checkout of the repo? https://github.com/codrsquad/portable-python/blob/main/portable-python.yml#L13

If not, the create a file portable-python.yml in the folder where you're running this from, with this content:

folders:
  logs: "{build}/logs"

[paulmromer]

Sure. I've attached all four just for completeness.

00-portable-python.log

03-cpython.log

01-xz.log

02-openssl.log

Thanks for your help on this error and for developing this application. It is extremely useful for what I'm trying to do -- create a Python application for MacOS that includes its own build of Python. There are other applications that are supposed to do this, but for my app it is better to have a full python build. In particular, the new process for signing code and notarizing it is very demanding so it helps to have the full control that portable-python offers. I've tried building a full version of cpython but seem to end up with a build that is not fully portable; openssl seems to be the problem child. I hope this helps explain my use case. Let me know if you want more detail.

[zsimic]

Yes, it is hard to get a fully "portable" python binary :) I tried using this one at first: https://github.com/indygreg/python-build-standalone

But couldn't get it to work, and it patches a lot of stuff (I prefer a simple build, with no patching if possible - because it is high maintenance, have to keep up at every single new python version etc) + I wanted to have tests, be able to run in debugger etc (hence: don't like bash... which all those builds usually are always using...)

The portable-ness is achieved using a linker technique, and then verified post-build (using tools like ldd and otool to inspect every single .lib or .so)

Should work for you, the logs look good, will have to compare and try to find what may be causing it this week-end (nothing stands out at first glance)

[paulmromer]

Thanks again. Here is the output from portable-python diagnostics:

pp-diagnostics-output.log

Let me know if there is any other testing you want me to do. For what it's worth, based on building cpython, the two options that might be relevant are:

--enable-universalsdk --with-universal-archs=universal2

Here are the pages at cpython that discusses these options:

https://github.com/python/cpython/blob/main/Mac/README.rst

https://github.com/python/cpython/blob/main/Mac/BuildScript/README.rst

[zsimic]

Took a look on M1. Turns out: 3.8.7 (from Dec 2020) indeed does not build for me either (same error as you). However, 3.8.12 (latest 3.8, from Aug 2021) builds without issue.

I think this is due to that old version simply not building on M1 at all (since M1 was released in Nov 2020).

Do you have a strong reason to use 3.8.7? I don't think there's much I can do to make it build (it would require specific patches trying to build old versions, which I'd like to avoid).

[paulmromer]

Your explanation makes sense about why 3.8.7 does not work.

It is totally reasonable for you to focus on recent builds. I can certainly work with one of them. I just confirmed that 3.9.7 and 3.10.0 both work as well so there are plenty to choose from. Sorry that I did not figure this out myself.

Thanks for your quick response.