newbie questions

[omarcostahamido]

Hi @shakfu What a great repo! I have some newbie questions if you don't mind me asking:

• is this mac os only?
• does it only work with homebrew python installation or vanilla python from python.org also works?
• does it actually embeds python into the external or is the external connecting to local python installation?
• could you add a simple list of dependencies to the readme?
• any reason why you can't provide already compiled max externals?
  • I know you probably don't want to populate the repo history with it but you could always add them to a release?
• does this work with python packages?
  • is it possible to provide an example that installs for example numpy and makes use of it?

Thank you for your attention and sorry for the long list of questions. Best, OCH

[shakfu]

Hi @omarcostahamido

Just pushed a fix for relocatable-pkg. It'll default to your python version now (not perfect, but should do for now).

Can you try make relocatable-pkg again please?

I'll test make shared-pkg and make framework-pkg as per your setup with 3.7.. I did test with 3.7 earlier but not in Documents so I may have missed something.

[shakfu]

@omarcostahamido

I'm getting the same errors. Building in a directory with a space is problematic (even the compiler can't handle it), see the error which brings down framework-pkg:

Making all in xz
gcc -DHAVE_CONFIG_H -I. -I../..  -DLOCALEDIR=\"/Users/sa/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/xz/share/locale\" -I../../src/common -I../../src/liblzma/api -I../../lib  -pthread -fvisibility=hidden -Wall -Wextra -Wvla -Wformat=2 -Winit-self -Wmissing-include-dirs -Wstrict-aliasing -Wfloat-equal -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Waggregate-return -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-declarations -Wmissing-noreturn -Wredundant-decls -g -O2 -MT xz-args.o -MD -MP -MF .deps/xz-args.Tpo -c -o xz-args.o `test -f 'args.c' || echo './'`args.c
clang: error: no such file or directory: '8/Packages/py-js/source/py/targets/build/lib/xz/share/locale"'

The solution is either move py-js out of Documents and symlink to it as i have to or to stay in Documents and compile elsewhere: i.e. change the build directory to /tmp/ or otherwise.

Even having fixed that, the issue then is that openssl.org seems to be down (at least to me) so I can't download the openssl dependency. I tried to access the site via a proxy and and it was accessible so it may be that recurrent attempts to download via an automated program will cause them to blacklist your ip or something!!

It's not the end of the world, I will give it some thought and see if a work-around is possible, possibly by putting the dependencies in the releases section on github, to be retrieved as a fallback if primary access is not possible.. Let's see.

All this for a couple of python3 externals :smile:

[omarcostahamido]

Hi @shakfu

I'm sorry to hear of all the trouble you're going through with this. The good side? It is only making the project more bullet proof?

re:openssl.org, some say it is down http://www.websitedown.info/openssl.org Others say otherwise. If I just type it in the browser it does load the page, though.

Best

[shakfu]

Hi @omarcostahamido

I managed to finally download openssl and sort out the dependency issue. I also created a fallback download of the dependencies in the release section.

in addition, I pushed a change to the main branch: the default build directory is now: $HOME/.build_pyjs

Then after making this change and using python3.7 and running from $HOME/Documents/Max 8/Packages/py-js as per your setup, I managed to compile shared-pkg, framework-pkg, and relocatable-pkg using the new build location without errors except one which was the same in all cases: a consistent codesigning error which is due to iCloud syncing with files in the ~/Documents folder while codesigning occurs.

/usr/bin/codesign --force --sign - --timestamp\=none $HOME/Documents/Max\ 8/Packages/py-js/externals/py.mxo
$HOME/Documents/Max 8/Packages/py-js/externals/py.mxo: resource fork, Finder information, or similar detritus not allowed
Command CodeSign failed with a nonzero exit code

This is a well known problem, but it's also, for me, the nail in the coffin for the idea of building py-js from $HOME/Documents/Max 8/Packages/py-js. I think it's just just a bad idea for all the reasons that are self-evident in this 100+ post exchange. I do not relish supporting the scenario of someone building py-js from $HOME/Documents/Max 8/Packages.

So my formal advice will be as follows:

When you are building py-js DO NOT build from $HOME/Documents/Max 8/Packages/py-js, just download or clone the repo to any location in your home directory which doesn't have a space in its path and which is not syncing to iCloud, and symlink to the Packages directory as follows.

For example if I download py-js to my $HOME/Downloads

$ git clone https://github.com/shakfu/py-js.git
$ ln -s ~/Downloads/py-js ~/Documents/Max 8/Packages/py-js

When you are done building and testing then you can copy or move the py-js directory to $HOME/Documents/Max 8/Packages/py-js or just grab the externals and use them in your own projects.

[shakfu]

Hi @omarcostahamido

Don't get me wrong, I definitely think this process has been fantastic for the project and for as you said making it bullet proof.

I'm very happy with the burst of progress that's been made, and I would be very happy that as a result of this progress, that you were able to compile without errors. But the last one is a real pain which I have encountered before (it's in my dev notes even) and (see here) I'm not about to ask users to turn off iCloud syncing so they can compile py-js :smile:

[omarcostahamido]

hey @shakfu

just download or clone the repo to any location in your home directory which doesn't have a space in its path and which is not syncing to iCloud, and symlink to the Packages directory as follows.

I understand your argument. And I will just add that, as I mentioned before, 1) a max package usually goes to the Documents/Max 8/Packages folder (source code included), 2) not all users are using iCloud (I am not), 3) using an alias or symlink can actually mess up Max in a way (I've had this problem for a couple years where max would freeze anytime I changed a file in the search path while max and google drive were running, it turns out that the fault were some alias files I had in the Max 8/Packages folder that linked to a place in, go figure, the Downloads folder! ) I'll just leave it at that here for anyone brave enough to go through this 100+ post exchange... 🏆

I'm very happy with the burst of progress that's been made, and I would be very happy that as a result of this progress, that you were able to compile without errors.

omg so much pressure now 😄 Well then:

• make relocatable-pkg downloads python 3.7.9 (nice, I'm on 3.7.3 but this is good) and builds 🎉 ! I didn't notice the ~/.build_pyjs folder though

  • relocatable-pkg.txt

• make shared-pkg does create the ~/.build_pyjs folder, but it ends here:

  270606 INFO : SharedPythonForPkgBuilder remove file: /Users/omar/.build_pyjs/lib/python-shared/lib/python3.7/site-packages
  270606 WARNING: SharedPythonForPkgBuilder file not found: /Users/omar/.build_pyjs/lib/python-shared/lib/python3.7/site-packages
  Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
  "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
  exec(code, run_globals)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 239, in <module>
  app.cmdline()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/cli.py", line 194, in cmdline
  options.func(self, options)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 184, in do_pyjs_shared_pkg
  self.ordered_dispatch('pyjs_shared_pkg', args)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 79, in ordered_dispatch
  getattr(builder, method)()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1394, in install
  builder.install()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 905, in install
  self.post_process()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1143, in post_process
  self.ziplib()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 853, in ziplib
  self.lib_dynload.rename(temp_lib_dynload)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1312, in rename
  self._accessor.rename(self, target)
  FileNotFoundError: [Errno 2] No such file or directory: '/Users/omar/.build_pyjs/lib/python-shared/lib/python3.7/lib-dynload' -> '/Users/omar/.build_pyjs/lib/python-shared/lib/lib-dynload'
  make: *** [shared-pkg] Error 1

• and make framework-pkg does all this close to the end

  • framework-pkg_ending.txt

[shakfu]

Hi @omarcostahamido

Thanks for your detailed reply as always. Your feedback is most helpful!

Let me clarify my view on some of the point you made:

I understand your argument. And I will just add that, as I mentioned before, 1) a max package usually goes to the Documents/Max 8/Packages folder (source code included),

That's correct, I'm not suggesting to go against this requirement which is where packages are supposed to be deployed, just to build somewhere else and then deploy to the packages directory as expected.

After thinking about my hard constraint a bit more (with the benefit of a good night's rest and a nice morning coffee), I have a few ideas which may or may not work to resolve this.. Let's see ...

2) not all users are using iCloud (I am not),

Fair enough, but it is natural to expect that other users (like me) do use iCloud.

3) using an alias or symlink can actually mess up Max in a way (I've had this problem for a couple years where max would freeze anytime I changed a file in the search path while max and google drive were running, it turns out that the fault were some alias files I had in the Max 8/Packages folder that linked to a place in, go figure, the Downloads folder! )

I haven't had any problems, my build (tmp) directory is in ~/Downloads/projects/py-js and the former is symlinked to ~/Documents/Max 8/Packages/py-js

But I think all of the above is moot if you understand that I am only suggesting to separating the place of building the externals or package from the place where they are to be ultimately deployed.

make relocatable-pkg downloads python 3.7.9 (nice, I'm on 3.7.3 but this is good) and builds 🎉 ! I didn't notice the ~/.build_pyjs folder though

Hooray!! :grin:

In the case of relocatable-pkg, there is no building of python from src, it is downloaded as .pkg from python.org, unpacked as Python.framework and made relocatable in the support folder. Then pyjs externals are built against this and are linked to this framework from the externals folder. Because you don't run iCloud, there were no codesign errors in the last step.

The errors for shared-pkg and framework-pkg are quite specific to each case. I just tested them and they work for me, but I'll take a closer look at these and see what could be giving you problems.

[omarcostahamido]

Hi @shakfu

Thank you for being available to discuss this and look into my errors.

I should also just note that the Google Drive + Max + alias/symlink to Downloads problem is specific to my system's configuration (and thus my fault only). In particular, not everyone is expected to use Google Drive, the same way not everyone is expected to use iCloud. This all just goes to say that things may be a bit more complex than they should be expected to be (with no apparent reason for it: Thx giant software companies!)

Looking forward to hearing what you discover next 😉

OCH

[shakfu]

Hi @omarcostahamido

I had some time recently to make a few improvements which I think will help going forward:

Reinforce the implementation of the external build directory $HOME/.build_pyjs

Building everything in one standard location has obvious benefits, so this is the direction we are going with right now.

It currently has the following subdirectories:

• downloads/ for caching the downloads of dependencies and Python source distributions
• src/ where building of build variants occurs
• lib/ where the static or dynamic library products of building are placed
• logs/<py_version>/ where full build logs are stored
• externals where the Xcode builds of externals are first placed before being copied to $HOME/Max 8/Packages/py-js/externals (not yet fully implemented)

Improve testing automation with full build-logs across python versions which seems required given this thread.

This was in response to attempting to read your sometimes truncated text files of your build errors. It seemed appropriate to standardize this step and automate across all build variants.

To build a specific build variant and save a a full build log to file (see above):

source source/py/scripts/funcs.sh
runlog shared-pkg

The above will run make shared-pkg and save all stdout and stderr to ~/build_pyjs/logs/<python_version>/shared-pkg.log

To build all build variants and save a full build log for each to file (see above):

source source/py/scripts/funcs.sh
runlog_all

or without Homebrew python (if you don't have homebrew)

runlog_all_no_brew

To check all logs for a given version

check_all

# and for those without homebrew

check_all_no_brew

I forgot to mention to that all logging automatically replaces the $HOME prefix with ~ to depersonalize the logging.

You can see my build logs for all build variants across python versions 3.7, 3.8, 3.9, and 3.10

logs.zip

If you don't mind, can you please do a full run for python 3.7 on your machine and share the logs in case you have build errors?

This would entail running this

source source/py/scripts/funcs.sh
runlog_all_no_brew 

and zipping up and sharing $HOME/.build_pyjs/logs/3.7.x

Thanks again for all your help!

[shakfu]

Hi @omarcostahamido

I have a branch ('dev') which is recently tested to work for your specific case / setup and has been tested for Pythons 3.7 - 3.10 without issue. The only thing you can't do is build the homebrew-specific variants homebrew-ext and homebrew-pkg. Everything else should work if you have Xcode installed.

Because of the recurring 'space-in-path` problem which even appeared during compilation of the python dependencies (xz, bzip2, and openssl), which were outside my ability to fix generally, I decided to apply what I believe is the safest approach possible to resolve this use case:

• Prohibit any compilation whatsoever in $HOME/Documents/Max 8/Packages/py-js

• All compilation and building for dependencies and externals is now exclusively done in $HOME/.build_pyjs and its subdirectories.

• The final python3 external products are then copied at the end of a successful compilation to $HOME/Documents/Max 8/Packages/py-js/externals

To get the dev branch do the following:

# in $HOME/Documents/Max 8/Packages
git clone https://github.com/shakfu/py-js.git
cd py-js
git fetch
git switch dev

Then test everything:

make test

This will run through a full test of building all of the variations, with the results and build times being displayed. Depending on your machine speed, these may take a while.

You can recall the results of the test at a later point as follows

make check

Let me know how it goes.

[omarcostahamido]

Hi @shakfu

Apologies, I am realizing now I missed your previous post, which I will address next.

The only thing you can't do is build the homebrew-specific variants homebrew-ext and homebrew-pkg.

🤣 oh the irony! hahaha...

• Prohibit any compilation whatsoever in $HOME/Documents/Max 8/Packages/py-js

Oooff! 🤐 Does it throw an error if you try to run just on that folder, or any folder with a space in its path? 😐 edit: ok, got it. Everything happens in $HOME/.build_pyjs now.

The final python3 external products are then copied at the end of a successful compilation to $HOME/Documents/Max 8/Packages/py-js/externals

Allow me to suggest that hardcoding this is a bad idea. Not everyone might have their Max folder in $HOME/Documents/. So, please don't write it as an absolute path, but a relative one: ./externals or ../../externals in case of running it from source/py.

[omarcostahamido]

If you don't mind, can you please do a full run for python 3.7 on your machine and share the logs in case you have build errors? This would entail running this

The console:

$  source source/py/scripts/funcs.sh
$  runlog_all_no_brew 

running 'make default'

real    0m8.230s
user    0m1.051s
sys 0m0.664s

./logs/3.7.3/default.log                       -> FAILURE: 0 out of 2 builds OK

running 'make shared-pkg'

real    2m31.926s
user    1m39.540s
sys 0m33.165s

./logs/3.7.3/shared-pkg.log                    -> FAILURE: 0 out of 2 builds OK

running 'make shared-ext'

real    0m19.772s
user    0m9.738s
sys 0m8.552s

./logs/3.7.3/shared-ext.log                    -> FAILURE: 0 out of 2 builds OK

running 'make static-ext'

real    0m2.043s
user    0m0.540s
sys 0m1.342s

./logs/3.7.3/static-ext.log                    -> FAILURE: 0 out of 2 builds OK

running 'make framework-pkg'

real    0m18.865s
user    0m9.392s
sys 0m8.087s

./logs/3.7.3/framework-pkg.log                 -> FAILURE: 0 out of 2 builds OK

running 'make framework-ext'

real    0m19.288s
user    0m9.508s
sys 0m8.154s

./logs/3.7.3/framework-ext.log                 -> FAILURE: 0 out of 2 builds OK

running 'make relocatable-pkg'

real    0m52.524s
user    0m21.513s
sys 0m17.813s

./logs/3.7.3/relocatable-pkg.log               -> SUCCESS: 2 out of 2 builds OK

and zipping up and sharing $HOME/.build_pyjs/logs/3.7.x

This build_pyjs folder was not created this time (yes i am seeing all the other invisible folders)

edit: from ./logs: 3.7.3.zip

[shakfu]

Hi @omarcostahamido

Please get the latest branch ('dev') as per my last post. You seem to be out of step with the posts. Please follow the steps in my last post, not the one before last :smile:

Allow me to suggest that hardcoding this is a bad idea. Not everyone might have their Max folder in $HOME/Documents/. So, please don't write it as an absolute path, but a relative one: ./externals or ../../externals in case of running it from source/py.

Nothing is hardcoded. There are no absolute paths used, only relative path. In any case, it's all configurable ultimately. This branch 'dev' is specifically to handle and test your use case.

The master branch is working as per the symlink-to-packages method I prefer, but one way of doing things can be changed via configuration to do it the other way.

[omarcostahamido]

You seem to be out of step with the posts.

hold on, I'm answering all the things in order. It takes time to do all the compilations here.

To get the dev branch do the following: Then test everything: make test

Here's what happened:

$  make test
>>> cleaning externals
>>> cleaning support directory
>>> cleaning build artifacts from local-sys target
>>> cleaning build artifacts from homebrew-pkg target
>>> cleaning build artifacts from homebrew-ext target
>>> cleaning build artifacts from framework-pkg target
>>> cleaning build artifacts from framework-ext target
>>> cleaning build artifacts from shared-pkg target
>>> cleaning build artifacts from shared-ext target
>>> cleaning build artifacts from static-pkg target
>>> cleaning build artifacts from static-ext target
>>> cleaning build artifacts from relocatable-pkg target
>>> cleaning build directory
>>> running tests

running 'make default'

real    0m4.782s
user    0m0.978s
sys 0m0.480s

~/.build_pyjs/logs/3.7.3/default.log           -> FAILURE: 1 out of 2 builds OK

running 'make homebrew-pkg'

real    0m6.468s
user    0m1.643s
sys 0m1.311s

~/.build_pyjs/logs/3.7.3/homebrew-pkg.log      -> FAILURE: 0 out of 2 builds OK

running 'make homebrew-ext'

real    0m5.562s
user    0m1.491s
sys 0m1.304s

~/.build_pyjs/logs/3.7.3/homebrew-ext.log      -> FAILURE: 0 out of 2 builds OK

running 'make shared-pkg'

real    4m20.318s
user    3m0.586s
sys 0m51.593s

~/.build_pyjs/logs/3.7.3/shared-pkg.log        -> FAILURE: 0 out of 2 builds OK

running 'make shared-ext'

real    1m56.238s
user    1m23.673s
sys 0m24.728s

~/.build_pyjs/logs/3.7.3/shared-ext.log        -> FAILURE: 0 out of 2 builds OK

running 'make static-ext'

real    2m5.724s
user    1m37.792s
sys 0m20.783s

~/.build_pyjs/logs/3.7.3/static-ext.log        -> FAILURE: 0 out of 2 builds OK

running 'make framework-pkg'

real    1m57.009s
user    1m21.525s
sys 0m24.765s

~/.build_pyjs/logs/3.7.3/framework-pkg.log     -> FAILURE: 0 out of 2 builds OK

running 'make framework-ext'

real    1m56.812s
user    1m24.074s
sys 0m24.973s

~/.build_pyjs/logs/3.7.3/framework-ext.log     -> FAILURE: 0 out of 2 builds OK

running 'make relocatable-pkg'

real    0m43.716s
user    0m18.873s
sys 0m14.810s

~/.build_pyjs/logs/3.7.3/relocatable-pkg.log   -> SUCCESS: 2 out of 2 builds OK

the log 3.7.3.zip

Best

[shakfu]

Hi @omarcostahamido

After reading through one of your logs, it looks like you are running things out of a virtualenv called MYENV. If so, that may be the problem, since the test environment which was used did not entail using a virtualenv.

Please see the attached log of my python3.7 run and where it differs from yours.

3.7.9.zip

[shakfu]

Hi @omarcostahamido

After spending more time with your logs, I am more and more convinced that we should put aside py-js for now and figure out if you have the basic dependencies and infrastructure to compile python3.7 on your system.

Just to highlight the 'breaking' errors in the simpler (-ext) variations:

static-ext

error: [Errno 2] No such file or directory: '~/.build_pyjs/lib/python-static/include/python3.7m/pyconfig.h'

This error typically indicates missing dependencies during compilation of python.

shared-ext

Same dependency error as static-ext

error: [Errno 2] No such file or directory: '~/.build_pyjs/lib/python-shared/include/python3.7m/pyconfig.h'

framework-ext

In this case, you are unable to build any of the python builtin extensions starting with _struct

building '_struct' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I. -IObjects -IInclude -IPython -I/usr/local/include -I~/Documents/Max 8/Packages/py-js/source/py/MYENV/include -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c _struct.c -o build/temp.macosx-10.9-x86_64-3.7/_struct.o
clang: error: no such file or directory: '_struct.c'
clang: error: no input files

Here I can see that you have possibly installed a virtualenv MYENV which is perhaps a complicating factor, and that you are running Mac os 10.9 (Mavericks) which may or may not be an issue.

So before I spend anymore time trying to solve issues on my front, I think we should establish whether your can build python3.7 from source on your system:

Here are basic steps test this:

1. Download Python from python.org

Please download https://www.python.org/ftp/python/3.7.13/Python-3.7.13.tgz

2. Build it

tar -xvf Python-3.7.13.tgz
cd Python-3.7.13
./configure
make

If you have the dependencies, the you should be able to build the above without errors and then run the built python interpreter without errors:

./python.exe

Can you please test this case.

Thanks.

[omarcostahamido]

Hi @shakfu

a ./python.exe did appear after running all of it. For the reference it ended here:

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_gdbm                 _hashlib              _ssl               
ossaudiodev           spwd                                     
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd                
time                                                           

Failed to build these modules:
_lzma                 _tkinter                                 

Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381

running build_scripts
creating build/scripts-3.7
copying and adjusting /Users/omar/Downloads/buildingpython/Python-3.7.13/Tools/scripts/pydoc3 -> build/scripts-3.7
copying and adjusting /Users/omar/Downloads/buildingpython/Python-3.7.13/Tools/scripts/idle3 -> build/scripts-3.7
copying and adjusting /Users/omar/Downloads/buildingpython/Python-3.7.13/Tools/scripts/2to3 -> build/scripts-3.7
copying and adjusting /Users/omar/Downloads/buildingpython/Python-3.7.13/Tools/scripts/pyvenv -> build/scripts-3.7
changing mode of build/scripts-3.7/pydoc3 from 644 to 755
changing mode of build/scripts-3.7/idle3 from 644 to 755
changing mode of build/scripts-3.7/2to3 from 644 to 755
changing mode of build/scripts-3.7/pyvenv from 644 to 755
renaming build/scripts-3.7/pydoc3 to build/scripts-3.7/pydoc3.7
renaming build/scripts-3.7/idle3 to build/scripts-3.7/idle3.7
renaming build/scripts-3.7/2to3 to build/scripts-3.7/2to3-3.7
renaming build/scripts-3.7/pyvenv to build/scripts-3.7/pyvenv-3.7
/usr/bin/install -c -m 644 ./Tools/gdb/libpython.py python.exe-gdb.py
gcc -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration   -I. -I./Include    -DPy_BUILD_CORE -o Programs/_testembed.o ./Programs/_testembed.c
gcc     -Wl,-stack_size,1000000  -framework CoreFoundation -o Programs/_testembed Programs/_testembed.o libpython3.7m.a -ldl  -framework CoreFoundation     
sed -e "s,@EXENAME@,/usr/local/bin/python3.7m," < ./Misc/python-config.in >python-config.py
LC_ALL=C sed -e 's,\$(\([A-Za-z0-9_]*\)),\$\{\1\},g' < Misc/python-config.sh >python-config

edit:

Here I can see that you have possibly installed a virtualenv MYENV which is perhaps a complicating factor, and that you are running Mac os 10.9 (Mavericks) which may or may not be an issue.

I am running a venv, correct. No I am on Catalina. Never had Mavericks.

[shakfu]

Thanks, @omarcostahamido

How are you setting up your virtualenv? Have you tried without a virtualenv?

[shakfu]

@omarcostahamido May I suggest that you try again without a virtualenv with the 'dev' branch:

make test

[omarcostahamido]

Hi @shakfu

How are you setting up your virtualenv? Have you tried without a virtualenv?

$ python3 -m venv MYENV $ source MYENV/bin/activate $ pip install --upgrade pip setuptools

that's about it.

May I suggest that you try again without a virtualenv with the 'dev' branch: make test

Done it on another machine I got access to running macOS 11.6.4 and python 3.7.3 (no virtual env). Cloned your dev branch, ran make test and...

11.6.4-python3.7.3-devbranch.zip

[omarcostahamido]

Hey @shakfu

So, I just realized that you created this folder on my Documents folder:

[shakfu]

Hi @omarcostahamido

Not sure I understand why you keep introducing new variables into the test: a virtualenv, or a new computer with a different setup than yours. Indeed the default log which you attached for the latter gives some indication that it doesn't even have Xcode installed. The only way this process is going to work is if one maintains consistent testing conditions.

Let's take a pause please. I'm trying to help you solve your specific case provided it remains your specific case. Clearly (from your use of a virtualenv and someone else's computer) you are worried about running test code in your Documents folder.

That's fine. I understand, but let's then forget about the 'dev' branch, and please switch to the master branch of py-js on your machine without a virtualenv and git clone it into your Downloads folder or something. then symlink from the py-js folder of the cloned rep to the $HOME/Documents/Max 8/Packages/py-js after ensuring that any prior install of py-js has been removed. We will build outside of Documents and then move py-js to $HOME/Documents/Max 8/Packages/py-js if the build is successful.

Please cd into ~/Downloads/py-js and make test and let me know how it goes.

[shakfu]

HI @omarcostahamido

As you perhaps gathered from my last post, I was a little bit frustrated with the last results on your machine and was expressing it in a manner which was, in retrospect, unfair to you. It really was no problem to run out of virtualenv or try another computer to get some additional information. It's on me if the code is not working!

In any case, I retested all versions again on my machine, and got positive results. I retested with a virtualenv and got the same. To remove the possibility that Homebrew on my machine was giving me a hidden advantage with respect to dependencies, I tried another machine (an older MacBook air, with Catalina) and removed all traces of Homebrew on it and got positive results. In all cases, I ran the dev branch inside $HOME/Documents/Max 8/Packages and builds were successful except for the homebrew-specific ones which were expected to fail on the external machine.

From one perspective, it is also positive outcome that you were able to compile python to the end without error. If this is reproducible, then the errors may be coming from some of my post-processing code (to shrink python basically), so another testing scenario could be that I create a vanilla case for you to test without any post-processing and to see if that works. I will do that shortly as its a low hanging fruit and quite easy to do.

Also taking a few steps back from trees, it should be unnecessary for you, and perhaps others having the same issues as you, to compile the externals and install xcode if we have automated builds which are producing and releasing build variants in a standardized way (via Github Actions for example). Then you just install the package and you are off. This is another area where a lot can be done.

So let's keep going.

[shakfu]

Hi @omarcostahamido

The 'vanilla' variations, mentioned in my previous post have been pushed to 'dev' branch.

These build variations have no cleaning or post-processing except for 'tests' removal and removal of *.pyc files. This alone shrinks each external from 150 MB to 37 MB

I'd appreciate if you can try both versions:

1. self-contained external

   make vanilla-ext

and

2. package structure

   make vanilla-pkg

I hope these work for you and thanks again for your help!

[omarcostahamido]

Hi @shakfu

I'm sorry the 11.6.4 tests were not helpful at all. I had limited time with that machine and totally forgot to check xcode 😞

so another testing scenario could be that I create a vanilla case for you to test without any post-processing and to see if that works. I will do that shortly as its a low hanging fruit and quite easy to do.

I'd like to test that yes. I'm baffled that you copied my setup and got different results! I really have no idea what could be going on here... should we check the xcode version? I'm running xcode 11.6 here.

(...) if we have automated builds which are producing and releasing build variants in a standardized way (via Github Actions for example).

Wait, what!? You can setup github actions to build max externals?? I am very ignorant about github actions but I would very much like to learn a little bit about it. I would think that you need a mac (or windows) system to compile objects for those systems... github actions run on a linux system, no? (I mean, I remember seeing somewhere that you can actually point a local machine to be the designated worker for your action, but at that point it becomes just another layer of local setup in this case..)

The 'vanilla' variations, mentioned in my previous post have been pushed to 'dev' branch.

Ok, here we go:

So, make vanilla-ext didn't write the log file and ended here:

136413 INFO : VanillaPythonForExtBuilder changing working dir to: /Users/omar/.build_pyjs/lib/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/__main__.py", line 265, in <module>
    app.cmdline()
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/cli.py", line 194, in cmdline
    options.func(self, options)
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/__main__.py", line 202, in do_pyjs_vanilla_ext
    self.ordered_dispatch('pyjs_vanilla_ext', args)
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/__main__.py", line 72, in ordered_dispatch
    getattr(builder, method)()
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/core.py", line 1519, in install
    builder.install()
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/core.py", line 1005, in install
    self.post_process()
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/core.py", line 1444, in post_process
    self.fix_python_exec_for_framework2()
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/core.py", line 1423, in fix_python_exec_for_framework2
    self.cmd.chdir(parent_dir)
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/core.py", line 287, in chdir
    os.chdir(path)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/omar/.build_pyjs/lib/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS'
make: *** [vanilla-ext] Error 1

Then make vanilla-pkghad a similar reaction:

95414 INFO : VanillaPythonForPkgBuilder changing working dir to: /Users/omar/.build_pyjs/lib/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/__main__.py", line 265, in <module>
    app.cmdline()
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/cli.py", line 194, in cmdline
    options.func(self, options)
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/__main__.py", line 207, in do_pyjs_vanilla_pkg
    self.ordered_dispatch('pyjs_vanilla_pkg', args)
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/__main__.py", line 72, in ordered_dispatch
    getattr(builder, method)()
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/core.py", line 1519, in install
    builder.install()
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/core.py", line 1005, in install
    self.post_process()
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/core.py", line 1498, in post_process
    self.fix_python_exec_for_pkg2()
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/core.py", line 1477, in fix_python_exec_for_pkg2
    self.cmd.chdir(parent_dir)
  File "/Users/omar/Documents/GitHub/py-js/source/py/builder/core.py", line 287, in chdir
    os.chdir(path)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/omar/.build_pyjs/lib/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS'
make: *** [vanilla-pkg] Error 1

hum...

edit: Just tried running make vanilla-ext outside of venv. Same results.

[shakfu]

Hi @omarcostahamido

Thanks for testing. No worries, even if it's not working right now, the information that you just provided helps towards localizing the problem.

I'd like to test that yes. I'm baffled that you copied my setup and got different results! I really have no idea what could be going on here... should we check the xcode version? I'm running xcode 11.6 here.

I'm also very curious why it hasn't been working on your system. I don't think it's an issue if you are running Xcode 11.6. I'm running Xcode 12.4 myself (on Catalina 10.15.7). I'm not sure that's the problem.

Wait, what!? You can setup github actions to build max externals?? I am very ignorant about github actions but I would very much like to learn a little bit about it. I would think that you need a mac (or windows) system to compile objects for those systems... github actions run on a linux system, no? (I mean, I remember seeing somewhere that you can actually point a local machine to be the designated worker for your action, but at that point it becomes just another layer of local setup in this case..)

I'm just starting to learn about Github actions. There's even an approved cycling74 external template which has a pretty decent Github action configuration (for macos/windows) see here

I'm going to have a go at getting this working. I thought the codesigning part would be a blocker but apparently there are GitHub actions which do this. So theoretically, I should be able to generate all of the variations (for mac and windows) on demand.

So, make vanilla-ext didn't write the log file and ended here:

Ok, my bad, I should have asked you to pull the latest 'dev' branch commit to get a fix (avoid downloading dependencies for vanilla builds) and also do this to get logging:

make test-vanilla-ext

I'll attach a log of a successful build of the above, logs shoud be in .build_pyjs

vanilla-ext.log.zip

also try the following:

make test-vanilla-pkg

Thanks!

[omarcostahamido]

Hi @shakfu

This is the playlist regarding github actions that I was checking out the other day.

I thought the codesigning part would be a blocker but apparently there are GitHub actions which do this. So theoretically, I should be able to generate all of the variations (for mac and windows) on demand.

🙌

Ok, my bad, I should have asked you to pull the latest 'dev' branch commit to get a fix (avoid downloading dependencies for vanilla builds) and also do this to get logging:

Ok, here it goes: vanilla-ext.log vanilla-pkg.log

Thank you for looking into this

[shakfu]

It's quite curious why your system is failing to compile the external. Let's just look at one case: vanilla-ext-3.7-diff-analysis.zip

I have attached three files, your failed log of compiling vanilla-ext-3.7.3, my successful log of compiling vanilla-ext-3.7.9 and a diff-analysis between the two, which I will quote partially here (note that the red lines are from your failed compilation and the green lines are from my successful one):

4205c4211
< Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
---
> Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/c++/4.2.1
4259c4265
< checking for -llvm-profdata... no
---
> checking for llvm-profdata... no
4309,4311c4315,4317
< checking libintl.h usability... no
< checking libintl.h presence... no
< checking for libintl.h... no
---
> checking libintl.h usability... yes
> checking libintl.h presence... yes
> checking for libintl.h... yes
4535c4541
< checking for textdomain in -lintl... no
---
> checking for textdomain in -lintl... yes
4544c4550,4551
< checking for pkg-config... no
---
> checking for pkg-config... /usr/local/bin/pkg-config
> checking pkg-config is at least version 0.9.0... yes
4547a4555

I think this is most likely where the failure occurs, your system does not have pkg-config, which is defined as:

pkg-config is a computer program that defines and supports a unified interface for querying installed libraries for the purpose of compiling software that depends on them. It allows programmers and installation scripts to work without explicit knowledge of detailed library path information. (wikipedia)

> checking for --with-decimal-contextvar... yes
4552c4560
< checking if PTHREAD_SCOPE_SYSTEM is supported... yes
---
> checking if PTHREAD_SCOPE_SYSTEM is supported... no
4572c4580
< checking for bind_textdomain_codeset... no
---
> checking for bind_textdomain_codeset... yes
4839,4840c4847,4848
< checking whether wchar_t is signed... yes
< checking whether wchar_t is usable... no
---
> checking whether wchar_t is signed... no
> checking whether wchar_t is usable... yes
4845c4853
< checking whether right shift extends the sign bit... yes
---
> checking whether right shift extends the sign bit... no
4896c4904
< checking for pkg-config... no
---
> checking for pkg-config... /usr/local/bin/pkg-config
4924c4932
< 47672 INFO : VanillaPythonForExtBuilder make altinstall
---
> 56596 INFO : VanillaPythonForExtBuilder make altinstall
5043c5051
< ./Modules/getpath.c:676:20: warning: 'NSModuleForSymbol' is deprecated: first deprecated in macOS 10.5 - dladdr() [-Wdeprecated-declarations]
---
> ./Modules/getpath.c:672:20: warning: 'NSModuleForSymbol' is deprecated: first deprecated in macOS 10.5 - dladdr() [-Wdeprecated-declarations]
5046c5054
< /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/mach-o/dyld.h:199:21: note: 'NSModuleForSymbol' has been explicitly marked deprecated here
---
> /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/mach-o/dyld.h:199:21: note: 'NSModuleForSymbol' has been explicitly marked deprecated here
5049c5057
< ./Modules/getpath.c:676:38: warning: 'NSLookupAndBindSymbol' is deprecated: first deprecated in macOS 10.4 - dlsym() [-Wdeprecated-declarations]
---
> ./Modules/getpath.c:672:38: warning: 'NSLookupAndBindSymbol' is deprecated: first deprecated in macOS 10.4 - dlsym() [-Wdeprecated-declarations]
5052c5060
< /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/mach-o/dyld.h:189:17: note: 'NSLookupAndBindSymbol' has been explicitly marked deprecated here
---
> /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/mach-o/dyld.h:189:17: note: 'NSLookupAndBindSymbol' has been explicitly marked deprecated here
5055c5063
< ./Modules/getpath.c:678:27: warning: 'NSLibraryNameForModule' is deprecated: first deprecated in macOS 10.5 [-Wdeprecated-declarations]
---
> ./Modules/getpath.c:674:27: warning: 'NSLibraryNameForModule' is deprecated: first deprecated in macOS 10.5 [-Wdeprecated-declarations]
5058c5066
< /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/mach-o/dyld.h:169:21: note: 'NSLibraryNameForModule' has been explicitly marked deprecated here
---
> /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/mach-o/dyld.h:169:21: note: 'NSLibraryNameForModule' has been explicitly marked deprecated here
5110c5118,5119
<       -framework CoreFoundation -ldl  -framework CoreFoundation;
---
>       -framework CoreFoundation -lintl -ldl  -framework CoreFoundation;
> ld: warning: dylib (/usr/local/lib/libintl.dylib) was built for newer macOS version (10.15) than being linked (10.13)
5121c5130
< sed 's/%VERSION%/'"`DYLD_FRAMEWORK_PATH=~/.build_pyjs/src/Python-3.7.3 ./python.exe -c 'import platform; print(platform.python_version())'`"'/g' < Mac/Resources/framework/Info.plist > ~/.build_pyjs/lib/Python.framework/Versions/3.7/Resources/Info.plist
---
> sed 's/%VERSION%/'"`DYLD_FRAMEWORK_PATH=~/.build_pyjs/src/Python-3.7.9 ./python.exe -c 'import platform; print(platform.python_version())'`"'/g' < Mac/Resources/framework/Info.plist > ~/.build_pyjs/lib/Python.framework/Versions/3.7/Resources/Info.plist
5129c5138,5139
< gcc     -Wl,-stack_size,1000000  -framework CoreFoundation Python.framework/Versions/3.7/Python -o python.exe Programs/python.o  -ldl  -framework CoreFoundation     
---
> gcc     -Wl,-stack_size,1000000  -framework CoreFoundation Python.framework/Versions/3.7/Python -o python.exe Programs/python.o  -lintl -ldl  -framework CoreFoundation     
> ld: warning: dylib (/usr/local/lib/libintl.dylib) was built for newer macOS version (10.15) than being linked (10.13)
5134c5144
< ./Tools/pythonw.c:214:9: warning: comparison of function 'posix_spawn' not equal to a null pointer is always true [-Wtautological-pointer-compare]
---
> ./Tools/pythonw.c:223:9: warning: comparison of function 'posix_spawn' not equal to a null pointer is always true [-Wtautological-pointer-compare]
5137c5147
< ./Tools/pythonw.c:214:9: note: prefix with the address-of operator to silence this warning
---
> ./Tools/pythonw.c:223:9: note: prefix with the address-of operator to silence this warning
5174c5184
< DYLD_FRAMEWORK_PATH=~/.build_pyjs/src/Python-3.7.3 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
---
> DYLD_FRAMEWORK_PATH=~/.build_pyjs/src/Python-3.7.9 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
5181c5191
< DYLD_FRAMEWORK_PATH=~/.build_pyjs/src/Python-3.7.3 CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup    ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall'  _TCLTK_INCLUDES='' _TCLTK_LIBS=''   ./python.exe -E ./setup.py  build
---
> DYLD_FRAMEWORK_PATH=~/.build_pyjs/src/Python-3.7.9 CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup    ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall'  _TCLTK_INCLUDES='' _TCLTK_LIBS=''   ./python.exe -E ./setup.py  build

Below you have the first sign of actual major failure: failing to build _struct.c

5185,5187c5195,5202
< gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I. -IObjects -IInclude -IPython -I/usr/local/include -I~/Documents/Max 8/Packages/py-js/source/py/MYENV/include -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c _struct.c -o build/temp.macosx-10.9-x86_64-3.7/_struct.o
< clang: error: no such file or directory: '_struct.c'
< clang: error: no input files
---
> creating build/temp.macosx-10.13-x86_64-3.7/Users
> creating build/temp.macosx-10.13-x86_64-3.7~
> creating build/temp.macosx-10.13-x86_64-3.7~/.build_pyjs
> creating build/temp.macosx-10.13-x86_64-3.7~/.build_pyjs/src
> creating build/temp.macosx-10.13-x86_64-3.7~/.build_pyjs/src/Python-3.7.9
> creating build/temp.macosx-10.13-x86_64-3.7~/.build_pyjs/src/Python-3.7.9/Modules
> gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include -I. -I/usr/local/include -I~/.build_pyjs/src/Python-3.7.9/Include -I~/.build_pyjs/src/Python-3.7.9 -c ~/.build_pyjs/src/Python-3.7.9/Modules/_struct.c -o build/temp.macosx-10.13-x86_64-3.7~/.build_pyjs/src/Python-3.7.9/Modules/_struct.o
> gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.13-x86_64-3.7~/.build_pyjs/src/Python-3.7.9/Modules/_struct.o -L/usr/local/lib -o build/lib.macosx-10.13-x86_64-3.7/_struct.cpython-37m-darwin.so
5189,5192c5204,5206
< creating build/temp.macosx-10.9-x86_64-3.7/_ctypes
< gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I. -IObjects -IInclude -IPython -I/usr/local/include -I~/Documents/Max 8/Packages/py-js/source/py/MYENV/include -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c _ctypes/_ctypes_test.c -o build/temp.macosx-10.9-x86_64-3.7/_ctypes/_ctypes_test.o
< clang: error: no such file or directory: '_ctypes/_ctypes_test.c'
< clang: error: no input files
---

After the failure to build _struct.c, there is the failure to build ctypes.c. Note that these are all python3 builtins (i.e. the source code for building them is included with python).

So my brief analysis leads me to speculate that the lack of pkg-config on your system as the most likely culprit.

See the pdf of the man page of pkg-config below:

pkg-config.pdf

My advice is have a look where the compilation is taking place in $HOME/.build_pyjs/src/Python-3.7.3 and see if you can find _struct.c etc.. also have a look why you don't have pkg-config installed and whether this is impactful to other programs which are trying to use to compile on your system.

[shakfu]

Hi @omarcostahamido

I have just run my first 'matrix'-type github workflow (i.e. macos 10.15 Catalina across pythons 3.7 to 3.10) and happily, I had a successful run on the first attempt which matches the testing I've been doing on my own machine.

The success of this run increases the likelihood that you are missing something or the other which is preventing your machine from building successfully. One thing occurred to me recently: have you tried to update your command-line tools? (see this stackoverflow article) which may help.

Incidentally, the successful run was running the 'master' branch, not the 'dev' branch which I have kept stable to test your particular setup preferences.

The GitHub workflow used was just a test run which didn't produce anything that one can use. In fact, it's really basic, but I am quite impressed how powerful this system can be:

name: Build2

on:
  workflow_dispatch:

jobs:
  build:
    name: macos-10.15 python-${{ matrix.python-version }}
    runs-on: macos-10.15
    strategy:
      matrix:
        python-version: ['3.7', '3.8', '3.9', '3.10']

    steps:
      - uses: actions/checkout@v3
        with:
          submodules: 'recursive'
          fetch-depth: '0'

      - name: setup python
        uses: actions/setup-python@v3
        with:
          python-version: ${{ matrix.python-version }}

      - name: Display Python version
        run: python3 --version

      - name: build_default
        run: make

      - name: build_shared_pkg
        run: make shared-pkg

      - name: build_shared_ext
        run: make shared-ext

      - name: build_static_ext
        run: make static-ext

      - name: build_framework_pkg
        run: make framework-pkg

      - name: build_framework_ext
        run: make framework-ext

      - name: build_relocatable_pkg
        run:  make relocatable-pkg

The next iteration will attempt to try to extract the build products, codesign them, notarize them, and package them for release / download.

Before I do that, I want to get Apple Silicon compatibility working. I took a step towards that on the master branch by updating the sdk to version 8.2 and exclusively using max-sdk-base. After doing this, I had a quick attempt at the beginning of the weekend to compile an arm64 version of framework-ext on my m1 MacBook air, and it worked after a few tweaks.

So this direction looks promising.

[omarcostahamido]

Hi @shakfu

I'm really enjoying this thread. Thank you so much for your careful look into the logs. So, first things first, if I run which pkg-config it doesn't return a thing, so it is safe to say that it is MIA. But why?

have you tried to update your command-line tools?

I remember updating the command line tools at some point yes. When I run softwareupdate --list it shows a sec updt for mac os and safari. I can try installing them after I cleanup some open tabs here. And see if that does something. A quick google search for how to install pkg-config shows several homebrew entries.

Also, wow! I'm also really impressed that workflow actions are working! This is really exciting. So, it ran all the make builds yet it didn't retrieve anything, so what did it do?

edit: note to self this is a nice starting place for git actions.

edit2: BTW, I was looking at the workflow and wondering if it wouldn't be better to split each build flavour into separate jobs so that they can run in parallel...

[omarcostahamido]

Hey @shakfu Plot Twist! I just ran xcode-select --install and it... started installing command line tools!? I don't understand this... I remember installing it before. And also, when I openned xcode to inspect preferences > locations it showed:

So, after installing command line tools (again?) I'm now trying again make test-vanilla-exton dev branch: vanilla-ext.log

I reckon which pkg-config still returns nada. So, you'll probably still find the conflict there around pkg-config?

[shakfu]

Hi @omarcostahamido,

if I run which pkg-config it doesn't return a thing, so it is safe to say that it is MIA. But why?

One thing for sure is that it isn't on your system, but it's also the case that it isn't installed by default by Xcode tools, so it may not even be required. I was hoping that an update to xcode-select tools would have have installed some missing dependencies. But it looks like you are still having issues.

Another option would be to install the homebrew base system (without installing homebrew python) which might help. In fact all of the macos build environments used by GitHub have homebrew installed (see macos-10.15 and macos-11

Also, wow! I'm also really impressed that workflow actions are working! This is really exciting. So, it ran all the make builds yet it didn't retrieve anything, so what did it do?

I'm still getting comfortable with the GitHub workflows and actions. I figured out how to cache and how to make available the externals at the end of a run, but I doubt these are usable until they are codesigned and notarized. This is possible but I haven't gotten round to it yet.

My current plan after trying a bunch of different configurations is to only use matrix-based multi-os and multi-python-version 'matrix'-based workflows for testing (without trying to collect the produced externals and linked python distributions). It's too complex to do this, and it isn't necessary to have one solution which fits all cases.

I am now testing a model which envisions one workflow per variant (using a simple matrix build to generate a package for each python versions). This can be extended in the future with an os dimension (for windows and arm64 architectures) to generate a package for os / python-version or even forgot about python-versions and just do iterate over os cases.

Incidentally, Github workflows don't actually support compiling on arm64 for macos yet.. so I can't use this system to test for apple silicon compatibility.

I think this will come though. Mostly, likely both GitHub (microsoft) and Apple have an interest in seeing this being made possible.

[omarcostahamido]

Hi @shakfu

In fact all of the macos build environments used by GitHub have homebrew installed (see macos-10.15 and macos-11

You always finding a way to rope me back into homebrew... haha

It's too complex to do this, and it isn't necessary to have one solution which fits all cases. I am now testing a model which envisions one workflow per variant (using a simple matrix build to generate a package for each python versions). This can be extended in the future with an os dimension (for windows and arm64 architectures) to generate a package for os / python-version or even forgot about python-versions and just do iterate over os cases.

I'd say your current goal is already pretty ambitious, but I like it!

Incidentally, Github workflows don't actually support compiling on arm64 for macos yet

So... you can focus on Windows first! Great! *wink

[shakfu]

Hi @omarcostahamido

Just fyi, I've implemented on-demand build (select any version of python3.7 to 3.10), with signing and notarization using GitHub workflows and actions. The workflow in question is called pyjs_builder and its implemented in pyjs_generic.yml

There's a caveat though, for *-pkg variants, the python framework or relocatable distribution that sites in the support folder isn't currently codesigned or notarized in this process (just the externals). Ideally they can codesigned and then notarized inside a .dmg but this isn't implemented by any GitHub action so far.. I could potentially do this manually, but I'd rather wait until there's an official GitHub action that does, which is inevitable.

Incidentally, I've been thinking that this extremely long exchange around one issue which you started. This has been great in many ways, but I think because of its length, it is perhaps not helpful for newbies who may get put off by the length and variety of topics discussed. It's probably better to close this and open up a bunch of topical issues which are shorter and more to the point. We can maybe have another issue called general development ideas or something for more general discussions and other one for for updates...

I have a couple more things I'd like to finish before trying to solve the compatibility problem (apple silicon or windows), namely producing a 'tiny' python3 external: I have it down to 8.8MB now, I want to keep going and see how far I can go before it becomes useless :smile:

[omarcostahamido]

Hi @shakfu

Congrats on pulling this off. And just like that I learned about actions artifacts, a neat way to store the output of running an action without having to make a release.

There's a caveat though, for *-pkg variants (...) Ideally they can codesigned and then notarized inside a .dmg

Are the pkg variants compiling a .dmg now? Or are you saying that packaging them in a dmg is the only way to notarize them? How does it differs from how it works now?

I've also been meaning to ask: in this table you mention that pkg variants have pip, but how do you actually use pip there? Is it possible to request a release asset?

Incidentally, I've been thinking that this extremely long exchange around one issue which you started. This has been great in many ways, but I think because of its length, it is perhaps not helpful for newbies who may get put off by the length and variety of topics discussed. It's probably better to close this and open up a bunch of topical issues which are shorter and more to the point. We can maybe have another issue called general development ideas or something for more general discussions and other one for for updates...

I agree. It is surely one of the longest github issue threads I've been on. And I'll remember it very dearly haha. So, if you must, please close this issue. It's been a great pleasure to have this open 1-on-1 conversation with you, Shakeeb.

I have a couple more things I'd like to finish before trying to solve the compatibility problem (apple silicon or windows),

Ha! One last thing, this reminded me of what you said previously:

Incidentally, Github workflows don't actually support compiling on arm64 for macos yet..

Since you moved sdk 8.2, shouldn't it be possible to compile a m1 compatible external on an intel-based mac? I'm pretty sure I've heard/read somewhere that is a thing.

[shakfu]

Hi @omarcostahamido

Let's conclude this thread with some good news.

There's a caveat though, for *-pkg variants, the python framework or relocatable distribution that sits in the support folder isn't currently codesigned or notarized in this process (just the externals). Ideally they can codesigned and then notarized inside a .dmg

Good news! This is now resolved, and one can generate on-demand both forms (self-contained externals and externals which depend on a Python.framework in the support folder of a package) from a GitHub action. Now you can select your variant, your python version, and then you have an artifact (a .zip archive) of a .dmg which is fully codesigned and notarized and which contains the py-js directory with externals and supporting patches and frameworks..

There was a hiccup along the way which prevented me from reporting this good news earlier, it seems that the shared-pkg format used in shared-pkg and homebrew-pkg is not a valid form that is acceptable to Apple's notarization algorithms (they only accept .frameworks). This was not a big problem and it just took a little bit to convert homebrew-pkg to .framework. shared-pkg itself is now not used (but the code still hasn't been deleted in case it proves useful for another platform).

I've also been meaning to ask: in this table you mention that pkg variants have pip, but how do you actually use pip there? Is it possible to request a release asset?

relocatable-pkg on its own includes pip by default. So you can just generate it using a GitHub action, and then when you are in the support directory of the package:

cd support
Python.framework/Versions/3.9/bin/pip3 install <something>

or since, you asked, I have uploaded the above and put it into 0.1 release section as a pre-release: py-js-relocatable-pkg-darwin-x86-3.9-5e7754b.zip

Since you moved sdk 8.2, shouldn't it be possible to compile a m1 compatible external on an intel-based mac? I'm pretty sure I've heard/read somewhere that is a thing.

I don't think i can do that on my intel mac running Xcode 12 on Catalina. No matter, I do have an M1 MacBook air which should help going forward.

[omarcostahamido]

Hi @shakfu

Great news indeed!

I very eagerly downloaded the py-js-relocatable-pkg-darwin asset and copied over the externals and support folder from the dmg file. When trying to install anything, though, I am faced with:

$  Versions/3.9/bin/pip3 install termcolor
Collecting termcolor
  Using cached termcolor-1.1.0.tar.gz (3.9 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

edit: and for the record

$  Versions/3.9/bin/pip3 list
Package    Version
---------- -------
pip        22.0.4
setuptools 58.1.0
wheel      0.37.1

[shakfu]

Hi @omarcostahamido

Preparing metadata (setup.py) ... error

That's what you get for playing with a pre-release :smile:

Ok, before I uploaded it, I pip installed numpy and it worked without issues. I was sufficiently impressed that I did not further testing.

It seems I may need to reduce the aggression on the shrinking I apply to relocatable: I was able to upgrade pip via pip3 install --upgrade pip and also upgrade setuptools via pip3 install --upgrade setuptools successfully. But then I got an error with termcolor (even though it was installed) related to ctypes which I have removed as part of my shrinking process. Strangely ctypes is not invoked in term color, so I suspected it's more to do with setuptools. I ran a dependency scan and it looks like setup tools does indeed invoke ctypes:

So that's the mystery of termcolor, or rather of setup tools needing ctypes which was removed.....By the way termcolor has no place in a python3 embedded in Max!!

I think the moral of the story is to release some kind of vanilla-relocatable-pkg which has minimal shrinking steps and maximum compatibility.

[omarcostahamido]

Hi @shakfu

By the way termcolor has no place in a python3 embedded in Max!!

yes, I know 😄 I just tried to install a couple different small python packages and they all failed with that same message.

[shakfu]

Hi @omarcostahamido

I have a fix for pip-bug in relocatable-pkg which I have just pushed. You can now even install termcolor without issues! Looks like it was due to the removal of ctypes after all -- a pretty weird dependency for an interpreted language!

Again, it's in the 0.1 release section under py-js-relocatable-pkg-darwin-x86-3.9-688a4ba.zip

Hoping this one actually works for you.

[omarcostahamido]

Hi @shakfu

Looks like it was due to the removal of ctypes after all -- a pretty weird dependency for an interpreted language!

haha 😆

Ok, let's see the new version: Oh no...

$  Versions/3.9/bin/pip3 install pitchtools
Collecting pitchtools
  Using cached pitchtools-1.5.tar.gz (13 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: pitchtools
  Building wheel for pitchtools (setup.py) ... done
  Created wheel for pitchtools: filename=pitchtools-1.5-py3-none-any.whl size=13087 sha256=64f64ce7d0a4b823f23106f9e817eb27a42b808933a34f7d002b43b197741140
  Stored in directory: /Users/omar/Library/Caches/pip/wheels/41/0d/56/c63ebf95e73004298965bf0eda2add8441b22b1ad55c8cd8ff
Successfully built pitchtools
Installing collected packages: pitchtools
Successfully installed pitchtools-1.5

It worked! 🎉

I'm a bit sad to say it, but this probably means that this thread is coming to a close, as you've completely addressed (in more ways than anyone could have anticipated) all the issues from the original post. This ability to work with the external packages was indeed the last point on that post!

So, I'm gonna miss our interactions here (*) but you should feel free to close this issue now. And pat yourself on the back: you did it!

Thank you so much. It's been a pleasure.

OCH

(*) unless you keep tagging me somewhere else to share some good news about this project 😄

[shakfu]

@omarcostahamido

That's great, Omar. I'm pleased it finally worked for you. Thank you very much for your feedback throughout. Tt definitely spurred me forward, and kept me on my toes :smile:

[omarcostahamido]

@shakfu RE your recent post at cycling 74 forum: I know you probably don't want to open another 100+ issue thread, but you can always reach me here on github with my handle @omarcostahamido :) it's easier to reply to github than to cycling74 forum, it always requires to open the page and login in... Best, OCH

[shakfu]

Thanks @omarcostahamido

If you have a specific question feel free to post a new issue to GitHub. If you want to chat about something in general. I'm on discord Max as 'shakfu'

[omarcostahamido]

I don't know that discord max. Can you send me an invite link?

[shakfu]

https://discord.gg/RmWQZHDc

[shakfu]

@omarcostahamido I just sent it... it's in the post above. It expires after use I think.