newbie questions

[omarcostahamido]

Hi @shakfu What a great repo! I have some newbie questions if you don't mind me asking:

• is this mac os only?
• does it only work with homebrew python installation or vanilla python from python.org also works?
• does it actually embeds python into the external or is the external connecting to local python installation?
• could you add a simple list of dependencies to the readme?
• any reason why you can't provide already compiled max externals?
  • I know you probably don't want to populate the repo history with it but you could always add them to a release?
• does this work with python packages?
  • is it possible to provide an example that installs for example numpy and makes use of it?

Thank you for your attention and sorry for the long list of questions. Best, OCH

[shakfu]

Thanks for your interest in this project. I suggest you go through the README carefully since it directly addresses many topics raised in your questions.

First, it's important to understand that this repo is a kind of umbrella project for different ways to use Python in Max/MSP. So in many cases, the answer is qualified by a 'it depends'.

• This project is macOS only for now and because of this it works by default with Homebrew installed python but it can also work with python compiled from source as well.

• The default build creates a lightweight external linked to your local homebrew python3; another variation embeds python3 into an external linked to python3 which resides in a Max package; and another variation embeds python into the external itself without an dependencies. There are other ways as well. The README gives an overview of the differences between the different approaches.

• The dependencies are there included in the README, but its not too complicated: python3 installed on your system, cython, Xcode or xcode-select --install command line tools for compilation.

• As per your request I have made a pre-release0 available (see the release section).

• It definitely works with python packages but depends on the build: if it connects to your local system python then you can use any package as you like. Otherwise, if you create a custom python build you can include packages manually or via pip.

Hope the above was useful.

[shakfu]

Hi @omarcostahamido just in case you were not aware I made a pre-release of a couple of python3 externals. Let me know if they work for you.

[omarcostahamido]

Max[33565:5524040] Error loading /Users/user/Documents/Max 8/Library/pyexternals/pyjs.mxo/Contents/MacOS/pyjs:  
dlopen(/Users/user/Documents/Max 8/Library/pyexternals/pyjs.mxo/Contents/MacOS/pyjs, 262): 
Library not loaded: /usr/local/opt/gettext/lib/libintl.8.dylib
Referenced from: /Users/user/Documents/Max 8/Library/pyexternals/pyjs.mxo/Contents/MacOS/pyjs

[shakfu]

Ok, it looks like there's still a dependency on gettext or libintl. This can be fixed but it will take a little time to revise the build script. Otherwise, a quick fix right now on a mac using homebrew should be:

brew install gettext

then try again.

Let me know if you still have problems.

[shakfu]

@omarcostahamido, I fixed the dependency issue on the pre-release0 externals (it's is now statically linked at build-time instead of dynamically linked so it should not require the brew install gettext solution mentioned above).

If you have a chance download the new externals from the releases section and give them a go. Let me know is you have any further issues.

[omarcostahamido]

Hi @shakfu py.mxo is good now, but pyjs still becomes red / unusable everytime. Nothing logged on the console tho :/ Thanks

[omarcostahamido]

btw, having a full python 3.9 system running in that little 9.3 MB external is a pretty impressive feat. I've previously compiled my python as an executable using pyinstaller for quick access (and sharing with non tech-avy users). If it is really possible to compile this guy with custom python packages included I am definitely gonna relearn this and follow your steps :)

[shakfu]

Hi @omarcostahamido

Good to hear that you go at least one of the externals working. It's my bad, I should have included a README in the release: the other one, pyjs.mxo is in fact a bit different frompy.mxo because it's a jsextension, a not well-documented way of making a c-based external which only works in the [js] object.

Unlike a regular c external such as py.mxo, a jsextension needs to reside in a Max package structure with some extra files to work properly:

If you look at the image of a minimal package with both externals. The pyjs.mxo external requires the jsextension folder with the maxclasswrap.js file inside. The package folder also needs a javascript folder to hold .js files which need to be used as an argument to the [js] object which hosts pyjs.mxo.

A demonstration of this is give below on the right side of the py_test_standalone.maxpat patcher file which tests both externals at the same time. If you open up this test file you can double-click on the [js] object as below and it will open up the test_pyjs.js file in the code editor. This is a demo on how to use pyjs.

Hope all of the above makes sense. My advice is to just use the py external for now. It has the most features, and then if you want to work in javascript and python at the same time then you can use the pyjs external. I will add a README to the release to make the above clearer in any case.

[shakfu]

btw, having a full python 3.9 system running in that little 9.3 MB external is a pretty impressive feat. I've previously compiled my python as an executable using pyinstaller for quick access (and sharing with non tech-avy users). If it is really possible to compile this guy with custom python packages included I am definitely gonna relearn this and follow your steps :)

(-: Yes, that was one of the biggest hurdles to making this feasible. In fact packaging was a huge pita which took 3 times as long as development due to a lot of trial and error as there's no official documentation to help you make a minimal python distribution and because Apple's code-signing and notarization rules kept classifying bundled externals which embedded python as illformed after compilation. The Aha moment was discovering that the only place you could keep the python standard lib was in the Resources folder in the bundle and that one needed to compile python in the most static way possible to get things down to a reasonable size and make Apple's codesigning / notarization algorithm happy -- a technique I learned from a side project of the beware briefcase project.

In any case, if you want to learn how a minimal python distribution is built have a look at the builder python package in the py directory.

[omarcostahamido]

Hi @shakfu ! I'm trying the new release 0.1 😄 Every time I open py_overview.maxpat this is printed to the console:

Should we be concerned?

On the readme you write

Open up any of the patch files in the patcher directory of the generated max package, and also look at the .maxhelp patcher to understand how the py and the pyjs objects work.

There is no pyjs help patch, and I think it would be good to have one since it appears as an object in the package but in reality we are never meant to create a pyjs object instance in a patch but only to call it from within a js max object - am I understanding it correctly?

edit: I just suggested a simple help patch on #4 this will be helpful if someone creates a pyjs object and starts scratching their head trying to understand how to use it. I think, for maxers, one of the first instincts is to alt-click to open the help patch 😉

[omarcostahamido]

I would also like to suggest moving all positional arguments into attributes. There are some options that I would like to try but I don't want to declare all the others before getting into that place. And I know that you can use any positional attribute and turn into a sort of attribute just by using its name but you need to know their exact name in order to do that. And if you do that you loose not only the neat autocomplete but also the handy instructions that let you quickly confirm what you are doing without the need to go open its reference page. For example:

edit: I just added #5 that address this

[shakfu]

Hi Omar,

Thanks for your feedback and for revisiting this project. I'll address a number of a your questions in the sequence that I read them:

1. Every time I open py_overview.maxpat there is a some debug information in the console. Should we be concerned?

It looks like I left @debug=on in this patch py_overview.maxpat.zip and it further must have cached some paths to related on my system in the patch. You should be able to switch it off by setting @debug=off (although, it seems some logging is not conditioned on the debug flag it seems). In any case, I have fixed this on the main branch (also attached in this post).

2. There is no pyjs help patch, and I think it would be good to have one.

An omission on my part. Couldn't agree with you more. The py-js/javascript/test_pyjs.js would be useful in this instance.

3. I would also like to suggest moving all positional arguments into attributes.

Agreed.

Thanks for the pull requests, Omar. Your feedback and contributions are much appreciated!

S

[omarcostahamido]

Dear Shakeeb,

My pleasure. I have my eyes on your repo, and I am really glad you continue to work on it. You really summed it up very well at the end of the readme the different attempts and struggles of dealing with Python in Max. I am personally very interested in helping you in this project (if you don't mind me asking some newbie questions along the way). Next thing up for me: figure out how to extend this with other python packages. I know you wrote in the readme:

each external 'bundle' contains an embedded python3 interpreter with a zipped standard library in the Resources folder which also has a site-packages directory for your own code.

But I am just thinking right now if there is a way to programmatically download a python package (we can't assume pip is available right?... or can py object actually provide it!? ). I'm going to try creating a virtual env, install numpy, rip the site-packages folder somewhere in the max path and add it to py using the pythonpath attribute, which I've been meaning to try.

Question: which one is this patch? If you also don't remember we can recreate it. It should be easily available to users when they open your package.

More serious question: do you plan on working on a windows version? I have access to a windows machine with visual studio. I can try dipping my toes here too. Reason for asking this ties back into what I said above about packages. Using the Resources folder is a neat trick but... it is so mac-centered: what happens when you have a windows version for this? It be nice to start thinking about a cross-platform way to go about this. (please correct me if I am missing something). Thx.

OCH

[omarcostahamido]

I think I need your insight into this. I still can't import python packages. Tried both playing around with pythonpath and copying packages to the py site-packages folder. I'm always faced with:

[py main] import numpy: ImportError('\n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy C-extensions failed. This error can happen for\nmany reasons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:\n\n https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n The Python version is: Python3.9 from "/Applications/Max.app/Contents/MacOS/Max"\n The NumPy version is: "1.21.5"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.\n\nOriginal error was: No module named \'numpy.core._multiarray_umath\'\n')

Now, for sake of completion I should note that I myself am running python 3.7.3 on purpose (no fancy homebrew install, just plain old install from python.org) and I wonder if that could influence the version of numpy that I install by default with pip to something that is not compatible with your embedded python 3.9.6 version (?). Thoughts?

edit: it seems this problem is specific to NumPy. I just tried another random package and it didn't complain when importing.

[omarcostahamido]

This issue suggested that a simple upgrade would suffice to solve this problem. I managed to install the very latest numpy available (and via pip in a machine that has python 3.8.9, not mine), but it still complains... What do you reckon?

[shakfu]

Hi Omar,

Not sure why, but I'm unable to push to this repo just recently and I've raised an issue with GitHub support to help me fix things so I'm pausing development till this happens.

In the meantime, I'll try to answer your questions as far as I can:

How to extend python with your own scripts and 3rd party libraries

1. The easiest solution is not to use a self-contained external and use an external that's linked to your system python3 installation. This is what gets built if you run ./build.sh in the root of the project. If you do it this way, you automatically get access to all of your python libraries.

2. If you insist on using a relocatable external (for packages or standalones), then it's a little bit more effort depending on what you want to do. First you have to note that there several locations where this can be done: A. The external site-packages: py-js/externals/py.mxo/Contents/Resources/lib/python3.9/site-packages B. The package script folder: py-js/examples/scripts C. Whichever path you set the patcher PYTHONPATH property to.

   For A, I have tested pure python scripts which should work without re-codesigning the externals, but if you add compiled extensions, then I think you have to re-codesign the external. Check out my maxutils project for help with that.

   For B, this is just a location that's automatically added to the PYTHONPATH, if I recall correctly, so pure python scripts and compiled extensions should be fine for Package Distribution. (code signing may be required for 3rd party distributions)

   For C, this is just setting that is done at the patch level so it should be straightforward.

   Note that it is possible to compile a python externals with an embedded pip (see the python builder package), but it will be rather large, but it will allow you to install to the bundle directly rather than manually.

Getting numpy to work in py-js

The easiest way is to use (1) above and just create an adhoc python external linked to your system python3 setup. If you have numpy installed there, then you should be good to go with the following caveat: the type translation system does not currently automatically cover native numpy dtypes so they would have to be converted to normal lists before use with Max. This is not a hard constraint, just not implemented yet.

If you want to go the way of (2) above, then it should be possible to package numpy in the full relocatable external version, but it will take some work. It'll be easiest to just start with the full external build (using the the python builder package) and not the minimal version that is built by default. If you do this, it's just straightforward and you can pip install numpy, but you will end up with a huge external bundle due to numpy's size (90.2 MB). I haven't yet tried getting access to numpy via setting PYTHONPATH property.

Windows version?

It should be theoretically possible to make the code more portable for use in windows, but I really gave up on windows development ages ago so it'll have to be someone other than me who make this possible.

Where is this patch?

Question: which one is this patch? If you also don't remember we can recreate it. It should be easily available to users when they open your package.

It the py_test_standalone.maxpat patch in the patchers folder.

S

[omarcostahamido]

Hi @shakfu

Not sure why, but I'm unable to push to this repo...

Strange... I use github desktop it works pretty well.

I really gave up on windows development ages ago...

Understood.

The easiest solution is not to use a self-contained external and use an external that's linked to your system python3 installation.

I understand that is the most stable approach, but for the people I work with they'll just loose interest the moment I ask them to open the terminal window or install python. So, yes, I really need the self contained version, which I also think is what makes your project so unique (and I haven't got into your other python implementations yet)

Getting numpy to work in py-js

I do want to go the route (2). I'm surprised you said:

It'll be easiest to just start with the full external build (using the the python builder package) and not the minimal version that is built by default.

In my mind the current minimal python external should have no problem accessing it using the PYTHONPATH property to add a directory where the 90mb numpy site-packages folders are located. In fact, that worked just fine to import some other packages just not numpy. Can you try this on your end? Using your current py object, add your site-packages folder (or a "new venv with just numpy"s site-packages folder) via pythonpath and then try to send it an import numpy message.

Thank you so much for your thorough response answering all my questions. I really appreciate it!

OCH

edit: I've noticed that you were able to merge the PRs (thank you), I wonder if you can overcome your issues the same way: creating a fork and sending PRs to here (silly workaround for a silly bug indeed).

[shakfu]

Hi @omar

With regards to your question about numpy, as you have seen it is not so straightforward to use it for some reason outside of method (1).

I will look into t a bit more. There may be a bug in relation the setting of pythonpaths or perhaps the minimal relocatable external is missing something or the other. That's why I had the sense that the regular full build would most likely work.

[shakfu]

Fyi Github support fixed the issue with the repo which looked like it was a malfunctioning server from their side. All good now.

[omarcostahamido]

With regards to your question about numpy, as you have seen it is not so straightforward to use it for some reason outside of method (1).

Indeed. But we must push through 😄 I got a hold of a machine where I was able to install python 3.9.6 (from python.org though) and create a virtual env and install numpy, then I grabbed the sitepackages folder and put it on my (3.7.3) venv folder aaaand...

it still gives me an error. But a different one! That means I am getting somewhere at least, even if in the wrong direction (ha!). Next thing I am going to try is to also copy over the 3 f2py files that I noticed are installed in the bin foder when numpy is installed. The problem is...

I will look into t a bit more. There may be a bug in relation the setting of pythonpaths or perhaps the minimal relocatable external is missing something or the other. That's why I had the sense that the regular full build would most likely work.

I have been having trouble with pythonpath method. The only way I can make it to work is by setting it on the object itself py @pythonpath {my_path_to_sitepackages}. If I just create a plain py object and then send it the message pythonpath {my_path_to_sitepackages} it doesn't really do anything... not even debug mode acknowledges it!

Thank you for also looking into this.

edit: yup right now this is the bottleneck I am facing. I think I tried all the combinations. putting these numpy site-packages folders inside the external resources doesn't help either.

[omarcostahamido]

Good to know Github got back to you and fixed problem! Thank you @github 🎉

[shakfu]

Incidentally, I created a 'fat' static python external and installed numpy in its site-packages and had the same error. After a little research, I found this which provided a good analysis (and possible solution) for why this happens with numpy. I did try a quick fix as per the advice but it still didn't work, so it'll probably be more involved to get statically compiled python externals to work. No quick fixes unfortunately!

In the meantime, use method (1) or ... use an alternative packaging variation (it's in the README):

In the root of the py-js directory:

make -C source/py bin-homebrew-pkg

or in py-js/source/py

make bin-homebrew-pkg

NOTE that this will create a py package in $HOME/Documents/Max 8/packages/py

In this package, you will find two externals in the externals folder, and in the support folder, you'll find a 'supporting' python installation. For the purposes of testing whether NumPy works in this case, do the following somewhere:

virtualenv venv
source venv/bin/activate
pip install numpy
mv venv/lib/python3.9/site-packages/numpy ~/Documents/Max 8/Packages/py/support/python3.9/lib/python3.9/site-packages/numpy

Then open ~/Documents/Max 8/Packages/py/patchers/tests/test_numpy.maxpat and after running the test you should get (what I got :-)

Note that recent changes in Max have allowed for this to work in standalones too. Just create your standalone application from a patcher which includes the py and pyjs objects. Once it is built into a then copy the whole aforementioned py package to <STANDALONE>/Contents/Resources/C74/packages and delete the redundant py.mxo in <STANDALONE>/Contents/Resources/C74/externals since it already exists in the just-copied package.

[omarcostahamido]

In the meantime, use method (1) or ... use an alternative packaging variation (it's in the README):

Ok, trying to use the make bin-homebrew-pkg but several errors occur:

/Documents/Max 8/Packages/py-js/source/py/api.c:6:10: fatal error: 'Python.h' file not found
#include "Python.h"
         ^~~~~~~~~~
1 error generated.

warning: Could not read serialized diagnostics file: Cannot Load File: Failed to open diagnostics file (in target 'py' from project 'py-js')

** BUILD FAILED **

The following build commands failed:
    CompileC /Users/omar/Documents/Max\ 8/Packages/py-js/source/py/targets/bin-homebrew-pkg/build/py-js.build/Development/py.build/Objects-normal/x86_64/api.o /Users/omar/Documents/Max\ 8/Packages/py-js/source/py/api.c normal x86_64 c com.apple.compilers.llvm.clang.1_0.compiler
(1 failure)
Command line invocation:
    /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -project targets/bin-homebrew-pkg/py-js.xcodeproj -target pyjs

/Documents/Max 8/Packages/py-js/source/py/pyjs.c:5:10: fatal error: 'Python.h' file not found
#include <Python.h>
         ^~~~~~~~~~
1 error generated.

** BUILD FAILED **

The following build commands failed:
    CompileC /Users/omar/Documents/Max\ 8/Packages/py-js/source/py/targets/bin-homebrew-pkg/build/py-js.build/Development/pyjs.build/Objects-normal/x86_64/pyjs.o /Users/omar/Documents/Max\ 8/Packages/py-js/source/py/pyjs.c normal x86_64 c com.apple.compilers.llvm.clang.1_0.compiler
(1 failure)
make: *** [build-bin-homebrew-pkg] Error 65

[omarcostahamido]

I have regular Python (from Python.org). And it is here /Library/Frameworks/Python.framework/Versions instead of the /usr/local/opt/python3/Frameworks/Python.framework/Versions. So, I tried changing line 64 of core.py but it is still complaining. Thoughts?

[omarcostahamido]

So, I know this is terribly messy, but I am sharing here in any case:

• I copied /Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m to /Documents/Max\ 8/Packages/py-js/source/py. I was really annoyed at the missing files and just wanted to get on with this.
• changed several file referring to #include "Python.h" and other neighbor files to be referring to the local copy #include "python3.7m/Python.h" (yeah, I told you it was messy)
  • I think these included py.h api.c api.h and pyjs.c
• changed the reference to a supposedly homebrew installed python location /usr/local/opt/python3/Frameworks/Python.framework/Versions to a vanilla python.org install location /Library/Frameworks/Python.framework/Versions
  • this was in core.py and bin-homebrew.sh
• changed a couple other places where I could still find a force reference to python 3.9
  • line 25 of /py-js/source/py/targets/src-shared-ext/py-js.xcodeproj/project.pbxproj
  • and in /py-js/source/py/targets/common.xcconfig
• only then it successfully compiled this guy... into two 2.9 and 2.6mb externals!?
  • (maybe you could clarify for me what was supposed to happen and what is the difference of this version?)
  • when I open /py-js/externals/py.mxo/Contents/Resources/lib it still shows a python3.9 folder and python39.zip file - fml 🤦
  • also, it did create a /Documents/Max\ 8/Packages/py folder but it is empty. It seems all it does will replace the contents of the current py-js package folder 😞
• finally I tried running and... it's weird. It kinda worked for a little bit. I realized that after I create a py object with the @pythonpath attribute declared in it, this path becomes available to other py objects... and then it crashed Max.

I think this all goes to say that I am really trying to get this working, but I am still not entirely sure how to properly use/configure it for my setup (python 3.7.3, installed via python.org installer). If you can make some sense out of this (and I am pretty sure you're the only person who can) it would be great. Thank you!

[shakfu]

Hi @omar

I think this all goes to say that I am really trying to get this working, but I am still not entirely sure how to properly use/configure it for my setup (python 3.7.3, installed via python.org installer). If you can make some sense out of this (and I am pretty sure you're the only person who can) it would be great.

Really really appreciate that you are trying to get this working. But first thing you should do is just take a breather and get some perspective on what should be done.

On MacOS, in this particular project, I spent the initial 25% of my time happily coding and solving programming problems, and the rest trying to overcome packaging problems (which include codesigning and notarization requirements by Apple). This is the reason why I have developed so many ways to package an external for my particular case (a Homebrew python user).

On MacOS Catalina (my current macOS version), there's system python2 (/usr/bin/python2), there's system python3 (/usr/bin/python3) and there's homebrew python3 (/usr/local/bin/python3 as a symlink to /usr/local//Cellar/python@3.9/3.9.10/bin/python3). As a rule, I don't touch my system python installations. So let's put these aside for now, and focus on my homebrew python3 installation.

All of the <xxx>-homebrew-<yyy> methods (in the Makefile in py-js/source/py/Makefile are specific to the category of homebrew installed python on macOS).

Please note that I did not test any of my packaging methods on a binary-installed python3 from python.org on a macOS. I have only developed externals that were built from source downloaded from python.org. Indeed, the python package py-js/source/py/builder handles this case by automatically downloading python source code from python.org, downloading its dependencies, configuring and compiling, etc..

It is a recipe for pain and suffering to try to retrofit one method on the other in the manner in which you were doing earlier. I know this too well (-:

In any case, it should not be too difficult to create a method to handle the case of someone who has installed python3 in binary form from python.org:

First step is to get the python3-config values:

PYTHON_VERSION=$(python3 --version | sed s/Python[[:space:]]//) # 3.9.5
PYTHON_VER=${PYTHON_VERSION%.*}          # 3.9
CFLAGS=`python3-config --cflags`
LDFLAGS=`python3-config --ldflags`

Please make sure that these are from you python.org installation and not the system pythons installation. Are you able to you obtain these?

[omarcostahamido]

Really really appreciate that you are trying to get this working.

This is a very exciting project, and I am also learning a lot from it, so thank you!

On MacOS Catalina (my current macOS version), there's system python2 (/usr/bin/python2), there's system python3 (/usr/bin/python3)

I am also on MacOS Catalina (10.15.7 more precisely)! I have a system python2 but not a system python3 (edit: correction, I do have a python3 in there, but whenever I do which python3 I get the one I installed)

All of the -homebrew- methods (in the Makefile in py-js/source/py/Makefile are specific to the category of homebrew installed python on macOS).

Ok, so I get that what you are trying to say is that I should create a new <xxx>-pythonorg-<yyy> instead of going around editing those that you have carefully crafted for homebrew version, haha. I understand the messy approach described on my last post is not the way to go. And I promise I won't create a PR with any of this trash! I am just trying to move in some direction here and learn how this works - or better yet, how this can work with my system 😉

Please note that I did not test any of my packaging methods on a binary-installed python3 from python.org on a macOS. I have only developed externals that were built from source downloaded from python.org. Indeed, the python package py-js/source/py/builder handles this case by automatically downloading python source code from python.org, downloading its dependencies, configuring and compiling, etc..

Ok, so this is the first thing I really need to clarify. The files I modified on the second bullet point of my last post (py.h api.c api.h and pyjs.c) are all referring to some Python header files. Is this the python source code you are referring to? If so, how can I actually make the builder be aware of their current location? With my python.org installation these files are all located under /Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m. What about on your side? I thought simply changing /usr/local/opt/python3/Frameworks/Python.framework/Versions to /Library/Frameworks/Python.framework/Versions would make the trick. You also say that it supposedly downloads python source from python.org, so this shouldn't even be a problem then? How can I make sure that it is in fact downloading these?

It is a recipe for pain and suffering to try to retrofit one method on the other in the manner in which you were doing earlier. I know this too well (-:

ooof! haha. no need to say more. I know, you know, we all know. But I am still learning this project though! Please have a little patience with me. And thank you for guiding me!

In any case, it should not be too difficult to create a method to handle the case of someone who has installed python3 in binary form from python.org: First step is to get the python3-config values:

Alright, here we go!

PYTHON_VERSION=$(python3 --version | sed s/Python[[:space:]]//)

• 3.7.3

PYTHON_VER=${PYTHON_VERSION%.*}

• 3.7

CFLAGS=python3-config --cflags

• -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g

LDFLAGS=python3-config --ldflags

• -L/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation

[shakfu]

Ok, with a little manual work, I managed to generate a 'sumo' version of the py external which contains 'numpy'. It took me a while lay it out internally and then to notarize it but I finally managed to sort that out, thankfully.

It's in the releases section, and you can also download it from here.

[omarcostahamido]

Interesting, the structure of this external is different from the previous ones. I tried running it and...

the plot thickens?

[shakfu]

Well this is classic Apple. It works, it was codesigned, and notarized, but then a manual verification after your observation shows that there's a problem:

$ spctl -a -vvv -t install py.mxo
py.mxo: rejected (invalid destination for symbolic link in bundle)
origin=Developer ID Application: XXXX (YYYYYY)

The only thing I can think of is the symlink of lib inside the Python.framework to the Resources folder in the bundle:

py.mxo/Contents/Frameworks/Python.framework/Versions/3.9/lib -> ../../../../Resources/Python/lib

or alternatively the symlink from libpython3.9.dylib to `Python'

py-js/externals/py.mxo/Contents/Resources/Python/lib/libpython3.9.dylib ->  ../../../Frameworks/Python.framework/Python

These are the only two symlinks that cross subfolders in the bundle. Mmm... looks like its only usable for me now till this is figured out.

Sorry, I thought I could hatch a quick solution for you.

Now you realize what a pain packaging is :smile:

[omarcostahamido]

I wish I could help more. (I guess what I can say is that I am here to try it as soon as you have a new breakthrough?) In the meantime, I am reviewing your post here.

I just tried strategy 2.B and it didn't work for python packages.

I wonder if it will be an easier (and more helpful) task, rather than figuring out how to codesigned and notarize numpy in particular, to fix the pythonpath issues in order to a) work with python packages located somewhere, or b) actually include a pip in the external that can download these packages.

In the meantime I think I found a fix for the several places where your hello.py script was cached and I will suggest a PR soon.

I would also like to propose removing the file /py-js/javascript/hello.py since it has the same name and can create a conflict with /py-js/examples/scripts/hello.py

Edit: here it is #6

[shakfu]

Hi @omar

Recent developments have been quite positive since we got the numpy case to work twice: first, using bin-homebrew-pkg, and then via my hastily assembled 'sumo' distribution (even if it did succumb in the end to a obscure bug). The latter should be fixable with more iterations and some robust codesign / notarization automations / checks so needs some time and the former is basically available today if you switch to Hombrew (-:

I also hear you about the pythonpath bug, which should be a low hanging fruit to fix (if it is indeed broken). I will have a look at this first when I can.

But before this, I'm going to pause accepting any non-trivial pull requests especially where there is a removal of files. Your current PR #6 assumes that these are cached but there is no writing to the package structure from the system, so these have been placed deliberately there by me previously -- in this case, perhaps to deal with testing the case for py and also pyjs externals.

Which brings me to another area where I could use some help, namely, test creation. On this topic, please see article and this repo in this regard.

In any case, appreciate your interest and feedback on this project.

[omarcostahamido]

Hi @shakfu (btw, everytime you write @omar you are tagging someone else haha, my username is @omarcostahamido)

Your current PR https://github.com/shakfu/py-js/pull/6 assumes that these are cached but there is no writing to the package structure from the system

I should have also been more clear in what I found/proposed. I just didn't remove all the instances of "/Volumes/VOLE/Users/sa/Documents/Max 8/Packages/py/examples/scripts/hello.py". For example, I left it on /patchers/tests/test_autoload.maxpat where it makes sense to be there since even the object is initialized with py @file hello.py @autoload 1. In fact, these changes are just that: clearing the file attribute contents on several max patches where the object or the context didn't seem to be referring to it anyway. And I should have separated the proposal to delete the second hello.py to another PR, yes 😞 . Admittedly I'm not 100% sure that /py-js/javascript/hello.py is not being used by anything. What I do know is that, given this is a max package now, everytime you have a message calling load hello.py, or a py object with @file hello.py, Max will complain that it has 2 files with the same name, and we need to be lucky that it will choose the correct one.

Which brings me to another area where I could use some help, namely, test creation. On this topic, please see article and this repo in this regard.

Thank you for sharing this, I will explore it too.

Have a great day/night.

OCH

[shakfu]

Hi @omarcostahamido

I have had another crack at the py external with numpy embedded which you can download here.

It may or may not work for you: I have codesigned it, notarized it successfully and tested it on 3 different macOS Catalina machines and it worked, but then on the 4th machine, with macOS Mojave, it did not load properly.

Let me know if it works for you as I recall you were running Catalina as well.

[omarcostahamido]

Hi @shakfu :)

Thank you for the new external. I tried opening but with no success:

Maybe the @loaderpath should have been translated into an actual path?

O

[shakfu]

Thanks @omarcostahamido

It's the same dynamic linking error I got on the 4th machine. Well, at least it's localized to libintl.dylib...

If you are at all interested in this area, this is not a bad overview.

[omarcostahamido]

Hey @shakfu ,

In the meantime I'm with a machine right now (BigSur11.1) where I was able to install brew and follow your instructions from the readme exactly as they are. But I was faced with:

py-js och$ ./build.sh
usage: dirname path
usage: dirname path
usage: dirname path
>>> cleaning build artifacts from bin-homebrew-sys target
>>> generate c code from cython extension
make: cython: No such file or directory
make: *** [build-extension] Error 1

Thoughts?

edit: maybe this will clarify it for me ;)

edit2: cython installed and...

xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance

time to install xcode then :/

[shakfu]

@omarcostahamido how did it go after you installed cython and xcode?

[shakfu]

Hi @omarcostahamido

If you have a chance, can you please try this new attempt to py-with-numpy

I have tried to disable the pesky dependency on libintl which is causing problems by configure during compilation.

If it doesn't work, please post the error as well.

Thanks for helping me out with testing!

[omarcostahamido]

@omarcostahamido how did it go after you installed cython and xcode?

it didn't. Apparently you can only install latest xcode from app store, and that version is not compatible with macos11.1. Then I went around looking for older versions, found this site, but it requires apple id login to download - and I shouldn't add my apple id there since it was not mine... I was able to install the upgrade to macos11.6.4 and then went to the app store aaaand.. it asked to log in to apple id. At that point I had already spent way too much time with this and had to leave. I will only have access to that machine again next week. I'll try to download xcode on my machine first and move it on a thumb drive or something... so much trouble haha.

If you have a chance, can you please try this new attempt to py-with-numpy

Of course, always! And yes, you did it! It imported numpy no problem. Now I have 3 questions to go from here: 1) how do I do this? Will you write a tutorial will all the steps required? Is it really just specific to numpy package in particular, or do you think it can easily be applied to other packages? 2) Now that you successfully solved this piece of the puzzle, do you reckon a py object with pip included would be both possible and an easier way to account for adding packages on the user's side instead of having to notarize them one by one and ship it in the external? 3) will it run on m1?

Thank you! OCH

[shakfu]

Hi @omarcostahamido

Happy it worked for you :smile:

I'll try to answer your questions:

How do I do this? Will you write a tutorial will all the steps required? Is it really just specific to numpy package in particular, or do you think it can easily be applied to other packages?

It's not specific to numpy, any python package can be installed.

1. Create external with embedded python3 (shared build)

cd py-js/source/py
python3 -m builder py_shared_ext -i && python3 -m builder shared_ext -i

You should find py.mxo and pyjs.mxo in pyjs/externals

2. Install required python packages

If you want to populate these externals internally with pure python packages, there should be no issue with codesigning. Just install them manually in the external's site-packages, for e.g py.mxo/Contents/Resources/lib/python3.9/site-packages.

If you want to install compiled extensions (like numpy), then do the same as above, but then you will need to re-codesign the external. If these are to be externally distributed, you will need notarize the external and you'll need an Apple Developer account (costs 100 USD / year)

This project should help you with codesigning externals (and standalones).

Now that you successfully solved this piece of the puzzle, do you reckon a py object with pip included would be both possible and an easier way to account for adding packages on the user's side instead of having to notarize them one by one and ship it in the external?

A. If you are the only user, then the easiest way is to just build the externals locally. Just do this from the root of the project:

./build.sh

Then you can pip install into your own system python3 site-packages and the externals will just work as expected.

B. The second easiest (and also portable) way is to embed the externals in a Max package which has a self-contained and relocatable python distribution in its support folder.

Currently, this requires a homebrew python3 installation. The following will create a py package in $HOME/Documents/Max 8/Packages/py:

cd py-js/source/py
make bin-homebrew-pkg

If you don't use homebrew, then it should be straightforward to do this from a source build and also include pip in the package's python distribution. I'll put this on the TODO list.

C. In the context of externals which embed a python3 distribution (shared or statically built), it would be pointless to include pip in such bundles as they are supposed to be sealed.

Nonetheless, It should be possible to include a step in the the bundle creation process to populate a bundle's site-packages as required. Again another item for the TODO list.

[shakfu]

Hi @omarcostahamido

A few updates which are related to your earlier requests:

• I have released both py and pyjs externals with a lite build of numpy which is less than 30MB compared to the full build which is 108MB!

• I've resolved the issues with the --enabled-shared pkg build and have updated it so that it contains pip. This really fits with your request for python3 externals which can be extended by both pure python and compiled extensions. To build this from python source, just do the following:

cd py-js/source/py
python3 -m builder pyjs_shared_pkg -i -b

The above will do two things:

1. create py.mxo and pyjs.mxo in py-js/externals
2. create a python3 distribution (with pip) in the py-js/support.

The externals are linked to the python3 distribution in support and the latter is self-contained and relocatable. So basically you have a package structure which can be extended and moved around without effecting codesigning or notarization.

I have hadn't yet had time to make a release for the above, but will probably do so at some point.

[omarcostahamido]

Wow! 👏 I gotta try this asap. Still restricted to compile with homebrew version, correct?

[shakfu]

Hi @omarcostahamido

Still restricted to compile with homebrew version, correct?

Actually, no. Homebrew python is only required in a few of the variations (see this pdf which covers all of the possible build variation.

The pyjs_shared_pkg variation I mentioned earlier is built from source automatically downloaded from python.org

[omarcostahamido]

So,

I just tried xperiment-py-with-numpy-lite-v0.1-x86-py.3.9.10.zip and it worked as advertised 😄

Then I very eagerly ran your builder command, and..

$  python3 -m builder pyjs_shared_pkg -i -b
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 24, in <module>
    from . import config
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/config.py", line 1, in <module>
    from .core import (Product, Recipe, PYTHON_VERSION_STRING,
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 318
    if (libs := self.product.libs_static) :
             ^
SyntaxError: invalid syntax

edit: I guess this operator is not valid before python 3.8. If I change it to == then...

.../Max 8/Packages/py-js/source/py/builder/depend.py", line 14, in <module>
    from termcolor import colored, cprint
ModuleNotFoundError: No module named 'termcolor'

and if I pip install termcolor then...

$  python3 -m builder pyjs_shared_pkg -i -b
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 24, in <module>
    from . import config
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/config.py", line 1, in <module>
    from .core import (Product, Recipe, PYTHON_VERSION_STRING,
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 192, in <module>
    class Product:
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 201, in Product
    url_template: str = None,
TypeError: 'type' object is not subscriptable

[shakfu]

Hi @omarcostahamido

Great to hear that the py-numpy-lite external works ok (-:

About the error with the builder command, that's my bad, a python walrus-operator slipped into the code somehow. I never use these, and it was the only instance.. Weird..

Anyway, I just pushed a commit to fix this or you can just do it yourself:

if (libs := self.product.libs_static):

to

libs = self.product.libs_static
if libs:

[omarcostahamido]

Hi @shakfu

Thank you for the quick fix. I've edited my old post but I should add it as a new post:

after fixing walrus-operator, I get..

.../Max 8/Packages/py-js/source/py/builder/depend.py", line 14, in <module>
    from termcolor import colored, cprint
ModuleNotFoundError: No module named 'termcolor'

and after installing termcolor, I am getting...

$  python3 -m builder pyjs_shared_pkg -i -b
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 24, in <module>
    from . import config
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/config.py", line 1, in <module>
    from .core import (Product, Recipe, PYTHON_VERSION_STRING,
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 192, in <module>
    class Product:
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 201, in Product
    url_template: str = None,
TypeError: 'type' object is not subscriptable

[shakfu]

Sorry about the termcolor dependency, it shouldn't be there... since the builder should run on vanilla python. The error you're seeing is basically due to my use of builtin gradual typing (see the libs_static parameter below) in python3.9? :

class Product:
    """A product of a builder."""

    def __init__(
        self,
        name: str,
        version: str,
        build_dir: str = None,
        libs_static: list[str] = None,
        url_template: str = None,
        **settings,
    ):
        self.name = name
        self.version = version
        self.build_dir = build_dir or self.name
        self.libs_static = libs_static or []
        #..

I could have written the following and it would have been more compatible with your version:

from typing import List

class Product:
    """A product of a builder."""

    def __init__(
        self,
        name: str,
        version: str,
        build_dir: str = None,
        libs_static: List[str] = None,
        url_template: str = None,
        **settings,
    ):
        self.name = name
        self.version = version
        self.build_dir = build_dir or self.name
        self.libs_static = libs_static or []
        #..

Is the typing available in python3.7?

[shakfu]

Hi @omarcostahamido

I've removed the list[xx] idiom and replaced by List[xx] and also remove the termcolor depedency.

Try out the latest commit when you have a chance.

Thanks.

[omarcostahamido]

Hi @shakfu

Is the typing available in python3.7?

Yes, it is!

Try out the latest commit when you have a chance.

Yes, I am doing it at the same time as you are, and noticed also the need to change lines 260 and 429. When I run it now I get...

$  python3 -m builder pyjs_shared_pkg -i -b
36   INFO : SharedPythonForPkgBuilder remove file: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.7.3
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 165, in <module>
    app.cmdline()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/cli.py", line 110, in cmdline
    options.func(self, options)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 139, in do_pyjs_shared_pkg
    self.ordered_dispatch(config.shared_pkg_builder, args)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 67, in ordered_dispatch
    getattr(builder, key)()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1047, in install
    builder.install()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 754, in install
    self.reset()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 377, in reset
    self.cmd.remove(self.src_path)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 166, in remove
    path.unlink(missing_ok=True)
TypeError: unlink() got an unexpected keyword argument 'missing_ok'

edit: I guess the reason is that, as per python documentation:

Changed in version 3.8: The missing_ok parameter was added.

edit 2: 🎉 50 comments in this thread now! 🎉