Apple silicon/ Blender 3.4.0 / MolNodes 2.0.2 package installation failure

[PlethoraChutney]

Hi Brady! Excited to play with the new features, and maybe finally dip my toes into contributing!

Unfortunately, trying to install the required packages using this button in blender:

results in an error message reading:

Failed to install required packages. Please check /Users/posert/Library/Application Support/Blender/3.4/scripts/addons/MolecularNodes/logs/side-packages-install.log

The contents of that log file are:

$ cat ~/Library/Application\ Support/Blender/3.4/scripts/addons/MolecularNodes/logs/side-packages-install.log
-----------------------------------
Installer Started: 2022-12-13 08:18:36.297157
-----------------------------------
###################################
Installer finished: 2022-12-13 08:19:05.242502
###################################

System Info

• M2 macbook air
• Monterey 12.6.1
• Blender 3.4.0
• MolNodes 2.0.2

Steps already taken

• Uninstall molnodes, restart blender, re-install molnodes

[BradyAJohnston]

How incredibly unhelpful of a log file haha. (also envoius of the M2).

Can you try launching Blender via the terminal blender or /Applications/Blender should launch it. Then when you try to install the packages again it should hopefully give a more helpful error message printed to the terminal.

[PlethoraChutney]

Let me tell ya, if I'd known how nice these machines are I would have "accidentally" destroyed my old Dell ages ago.

Ah, good call. It seems Blender is failing to build a wheel for numpy during the biotite install. I'll poke around a bit! Thanks!

[BradyAJohnston]

Are you getting the same compiler error as #102 where it's lacking gcc?

[PlethoraChutney]

No, I'm missing python.h. Trying to figure out why, since I should have it. I'm now just failing to install MDAnalysis, I got numpy working.

[BradyAJohnston]

Okay this is a good test case. This seemed to be an issue with earlier versions of Blender when I was originally developing the addon which is why I had the complicated installation od MDAnalsysis in an external python and you had to then link it to Blender's python.

It might be that it's the ARM build of Blender doesn't include the python.h file but it's now included in other versions?

[PlethoraChutney]

That was my thought as well, but I'm also failing to install MDAnalysis in my machine's own python, as well as conda! So I don't think I can blame Blender unfortunately haha. I'm getting close to just nuking all the pythons on my machine and starting over

[BradyAJohnston]

There comes a time in every computers life when one must remove any trace of the word python and start again

[PlethoraChutney]

taps plays

[BradyAJohnston]

🫡

[Luc1100]

I'm on an M1 Pro, Ventura 13.1, Blender 3.4.0, MolNodes 2.0.2 and I am having the same issue SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel. However, I'm not having any issues downloading MDAnalysis on my machine's own python.

[BradyAJohnston]

You could likely solve the issue immediately with the solutions in #24 but I'm more interested as to why it installs OK on windows / intel macs but not on ARM macs. Potentially a Blender distro issue

[Luc1100]

Soft linking a conda environment with the packages installed worked, hopefully you can get the one click install working soon!

[BradyAJohnston]

If it's an issue with the ARM compiled Blender then I may have to include a second-step of the install only for M1/M2 macs which copies how it was achieved in the previous version. It's not ideal as it significantly complicates the installation process unless somebody can figure something else out.

@Luc1100 can you share the exact steps that you followed to get it working? Does it continue to work after you restart Blender? Are you able to use the Molecular Dynamics import panel or just the PDB / Local File import?

[Luc1100]

To get it working I made a new conda environment conda create --name blender python==3.10 biotite==0.35.0 mdanalysis==2.2.0 -c conda-forge -y

Found the path to the environment /path/to/miniconda3/envs/blender

In /Applications/Blender.app/Contents/Resources/3.4, I renamed the python that Blender came with mv python _python

Then I made a soft-link to the anaconda version of python ln -s /path/to/miniconda3/envs/blender python

It has worked after every blender restart and the molecular dynamics import panel is there. The only issue that I am running into is probably user error (I import a trajectory and I can only see the points in wireframe mode regardless of the render engine)

[PlethoraChutney]

This works for me as well!

I am still but a lowly pip user --- using conda worked right away. I noticed that Blender is also using pip. Perhaps that's the issue.

Maybe this will finally get me to learn what the hell conda is.

[maxewilkinson]

I'm having the same problem, but @Luc1100 fix of linking a conda environment doesn't work for me. When I do this, Blender crashes upon startup. Hmm any ideas for other work arounds?

Using MacBook Pro with M1 Max, Monterey 12.5, Blender 3.4.1, MolNodes 2.0.2

[maxewilkinson]

It looks like my Blender was crashing because the conda environment was using x86_64 (via Rosetta?) instead of arm64. So Blender was trying to call arm-compiled python packages and getting x86 packages instead. I don't know if this is just a peculiarity of how I installed conda.

Anyway, to fix this, when you create the conda environment, use:

CONDA_SUBDIR=osx-arm64 conda create -n molnodes python==3.10 biotite==0.35.0 mdanalysis==2.2.0 -c conda-forge -y cd /Applications/Blender-3.4.app/Contents/Resources/3.4 mv python old_python ln -s /Users/maxwilkinson/opt/anaconda3/envs/molnodes python

Molecular Nodes now installs, but failed to import structures, because conda installed numpy 1.24.0, in which np.int used by Molecular Nodes is deprecated. This is then fixed by

conda activate molnodes CONDA_SUBDIR=osx-arm64 conda install numpy=1.22

So far this is working for me!

[BradyAJohnston]

Hi @maxewilkinson thanks for the bug report! Apple M1 / M2 continues to be a source of trouble, but I appreciate the troubleshooting you did and the fix that you posted.

I'll have a look into the np.int as that will be problematic.

[Aseamann]

Max's comment is what worked for me. I believe it ensures what gcc complier is being used?

https://github.com/BradyAJohnston/MolecularNodes/issues/108#issuecomment-1365540371

[BradyAJohnston]

Thanks @Aseamann for the details. Might require some more specifics for installation on M1 macs. Unfortunately I don't have a machine to test on, and I'm not sure what the approach would be to try and help with this from the package perspective.

[PlethoraChutney]

Here's a gist for the fix, which might come in handy if you're giving a workshop on MolNodes

https://gist.github.com/PlethoraChutney/fd34e3137044cb1e1efc881157e0e17b

[BradyAJohnston]

many thanks! I have a workshop coming up in a week and this will likely come in handy

[YaoYinYing]

Here is an optimized solution of MN package installation for Mac on M1/M2 Apple Silicon :

1. Make sure your Blender is built on apple silicon version and MN has already installed.
2. Check if you have already install python@3.10 via conda or homebrew: which python3.10 or brew list python@3.10. If yes, go to step 6.
3. Install miniconda or anaconda on ARM version.
4. Create a new conda env containing python3.10: conda create -n py310 python=3.10 -y
5. Activate it: conda activate py310
6. walk to the build-in python binary dir of blender: cd /Applications/Blender.app/Contents/Resources/3.6/python/bin
7. run pip with a CPATH anoucement:

   CPATH=$CPATH:$(python3.10-config --include | sed  -E 's/ ?-I/:/g') ../python3.10 -m pip install MDAnalysis==2.5.0

   To install the exactly pinned version of required packages in requirements.txt, please use the following instead:

   CPATH=$CPATH:$(python3.10-config --include | sed  -E 's/ ?-I/:/g') ./python3.10 -m pip install -r "/Users/$(whoami)/Library/Application Support/Blender/3.6/scripts/addons/MolecularNodes/requirements.txt"

---

Update: fix ./pip3.10 with ./python3.10 -m pip and Blender version to 3.6.

[caitlinmonney]

Sadly, I am in the same situation and I have not yet found a solution. Sam install package button with same error message. Have tried to install miniconda, no work Using Mac Studio Apple M1 Max , Blender 3.4.1 , MolNodes 2.3.1 Please help

[BradyAJohnston]

Hi @caitlinmonney, my recommendation would be to completely uninstall the Molecular Nodes addon and completely uninstall Blender.

Once you have a clean system, reinstall Blender. Before you install Molecular Nodes, use the script posted by @PlethoraChutney to build the required python libraries. Once that is complete, then try installing Molecular Nodes and see how that goes.

[PlethoraChutney]

Unfortunately someone I was helping who had an M1 chip was not able to resolve the issue --- the packages failed to install using Blender, and the workaround caused a crash-on-launch. We still haven't been able to figure out a fix (and I'll admit, I'm a bit confused how something that works on an M2 machine fails on an M1 machine).

[YaoYinYing]

Hi @caitlinmonney I got a slighter solution of more elegance at https://github.com/BradyAJohnston/MolecularNodes/issues/108#issuecomment-1429384983 by borrowing the Python.h from conda installed python3.10. Would you mind taking a try on that ?

And @BradyAJohnston I think this might work for most common cases.

[BradyAJohnston]

@PlethoraChutney I think I helped someone with an issue where Blender would crash on startup. Try reinstalling Blender, MN & the conda environment which seemed to fix the issue for them.

[BradyAJohnston]

@YaoYinYing would there be a way we could generalise this / make it simpler for users? If it works reliably can we enable this better on the MN installation side of things?

[BradyAJohnston]

I don't actually know much about C header files, but would we be able to just ship MN with our own python.h file?

I've had a look at the python.h that was included with my Intel Mac Blender, could we just include some version of this? https://gist.github.com/BradyAJohnston/09737a5eb30e30f79e46795cb942ab7d

This was in my Blender.app file at this location:

./Contents/Resources/3.4/python/lib/python3.10/site-packages/debugpy/_vendored/pydevd/pydevd_attach_to_process/common/python.h

[YaoYinYing]

@BradyAJohnston Well, I don't know much about the C header either. It looks like the build-in python has truncated quite a few things. I am working in progress on this automatical script based on the following steps:

1. determine whether the platform is macos on ARM;
2. check if conda or homebrew exists ;
3. check if python3.10 on ARM is installed via conda or homebrew;
4. if the desired python3.10 does not exists, run conda install or brew install (if brew install fails try reinstall from source building);
5. call python3.10-config --include and replace ?-I with ':' and store the result as HEADER_INCLUDE_DIR;
6. run 'CPATH=$HEADER_INCLUDE_DIR:$CPATH install -r requirements.txt'

Here is a phenotype script generated by ChatGPT: https://gist.github.com/YaoYinYing/8d6c297c0af30087bbefd0b2e73e47c9 I haven't carefully checked it but this idea might be feasible.

[YaoYinYing]

Also , the guideline at https://github.com/BradyAJohnston/MolecularNodes/issues/108#issuecomment-1429384983 has been updated to support brewed python3.10.

See logs at https://gist.github.com/YaoYinYing/36b793fbb751fbceb320ee9437a6d77b.

[YaoYinYing]

I'm not sure how the python.h at ./Contents/Resources/3.4/python/lib/python3.10/site-packages/debugpy/_vendored/pydevd/pydevd_attach_to_process/common works for Blender. In my mind, C & Python are case-sensitive. So I guess the 'Python.h not found' will still occur if not external CPATH is anounced before pip install.

[BradyAJohnston]

We could also switch back to creating the conda environment, then just adding that to Blender's python $PATH which was a little finicky, at least it didn't crash all of Blender when it didn't work.

[YaoYinYing]

@BradyAJohnston I think i have solved this one-click installation via conda environment. Feel so excited! https://gist.github.com/YaoYinYing/00f86f2cc400c7e9db335f6211d512f2

I have send a new PR for this fix. If you too think this solution is okay, please feel free to edit it and do some tests for hunting potential bugs.

[caitlinmonney]

First of all, thank you all very very much for being so proactive and trying to help out. I really appreciate it.

OK, so I have tried uninstalling and reinstalling everything in the suggested sequence. Still no success, I am not given the option to install atomise from the Add-ons install menu-as seen in our YT video-, I do get the Install Packages button and the error message as mentioned above.

With the help of @marwan-abdellah we have tried
pip install biotite==0.35.0 MDAnalysis==2.2.0
But no success Tried to uninstall the default numpy and then let the deps pull the appropriate version But still no success

I have tried the latest MN version with the latest Blender release on my MacBook Air Intel i5, all seems to work. Making my was through your tutorials to see if anything comes up. But I would very much like to find a fix for my M1 as this is my main machine. We-re going to continue to trouble/shoot on our end and cross our toes we find a fix. Let us know if you have further suggestions...

Thanks

[YaoYinYing]

Finally this happens to me again - I have no idea why it just works on my machine lol

Maybe I should visit the apple store nearby and look for the ARM Macs for more tests ....

[YaoYinYing]

@caitlinmonney Hi I just wrote a script for this issue. Would you please help me for a test? It would be fast.

You should do the following:

1. Remove Blender.app
2. Remove the user data of Blender (it would be at "~/Library/Application Support/Blender"): rm -r ~/Library/Application\ Support/Blender
3. Reinstall the latest Blender
4. Launch Blender, install the latest MN and enable MN
5. Open a terminal window and execute curl -fsSL https://raw.githubusercontent.com/YaoYinYing/MolecularNodes/ARM-mac-installer/MolecularNodes/troubleshoot/InstallDependencies_M1.sh | $SHELL to help you install the dependencies. If it fails and the X-code pops out for installing gcc, just agree and wait until it's done, then repeat this step once.
6. check the MN to see if you can load a PDB.

---

The workflow of this script:

1. Check if Blender and MN are installed. If not, exit 1.
2. Check if system-wide Python3.10 is installed. If so, go to step 6.
3. If no Python3.10 is detected, check if conda exists. If so, go to step 5
4. If no conda exists, download a miniconda pkg and install miniconda and apply it to shell
5. Setup a new conda env called MN_PY310 and activate it.
6. Run the bundled python for pip update pip and pip install stuffs with CPATH anouncement (for borrowing Python.h from MN_PY310)

---

Updates:

It works to M2 too.

[BradyAJohnston]

Maybe I should visit the apple store nearby and look for the ARM Macs for more tests ....

If somebody wants to bankroll an M1 / M2 mac for me to debug on, that would certainly be appreciated haha.

While this can potentially be a workaround for people, I don't like the idea of doing so much installation via the 'Install Packages' button etc. I'm chatting with some of the Blender devs about it. Hopefully I can get some more information about what the best way going forward might be.

[BradyAJohnston]

@caitlinmonney please do let us know any solutions that you might come up with. I'm pretty stuck with possible solutions. Without having access to an M1 / M2 machine to test & debug on I can't really be of much help unfortunately, and it's up to others with ARM machines to share possible solutions they have found that work for them.

[YaoYinYing]

I don't like the idea of doing so much installation via the 'Install Packages' button etc.

Me too. This has done so much unnecessary stuff for this issue. I will be very appreciated if someone in Blender dev team get python-dev inside this bundled Python exec.

[BradyAJohnston]

I have tried the latest MN version with the latest Blender release on my MacBook Air Intel i5, all seems to work. Making my was through your tutorials to see if anything comes up. But I would very much like to find a fix for my M1 as this is my main machine. We-re going to continue to trouble/shoot on our end and cross our toes we find a fix. Let us know if you have further suggestions...

@caitlinmonney and others who are able to get it installing on another machine. As a workaround, once you have the molecules imported, you can then open the .blend files on your main machine. The python packages are only required for the initial import step. All of the nodes should still work on your M1 / M2 machines without the need for the external python packages at all, as they are only required for importing.

[BradyAJohnston]

I've been chatting with some Blender devs, and it seems that it's likely not a Python.h problem, as Blender shouldn't be shipping with the required files for compiling packages.

What is likely causing the discrepancy between systems, is the lack of .whl that is pre-built on PyPi for M1 / M2 macs. Looking at the MDAnalysis PyPi page, they are missing an arm64 version which packages like SciPy have available. This causes the issue on M1 / M2 macs when they have to try and compile the packages themselves but don't have the pre-requisite files. This would also explain why earlier versions of Molecular Nodes had to handle package installation differently, when I was installing the most recent versions which may not have had the necessary .whl files and caused issues on more systems, whereas now it is limited to ARM macs.

MDAnlysis PyPi v2.2.0

MDAnlysis PyPi v2.4.2

SciPy PyPi

A solution?

I've chatted with the MDAnalysis devs on a few issues before, so I might be able to see if they can also release an ARM build of MDAnalysis which would potentially alleviate the installation issues. I don't really know much about python releases so I don't know if this is something that has to be done by them or can be built by PyPi somehow? Could we (somebody with an ARM mac) build a version or MDAnalysis for ARM macs, that we could include for installation on M1 / M2 macs which might solve the issues?

Anyone with some more information or knowledge about the whole system works please do let me know.

[BradyAJohnston]

I've contacted the MDAnlaysis devs and they are going to try and set up MacOS ARM builds on PyPi for the 2.5.0 release which should hopefully resolve this issue!! (https://github.com/MDAnalysis/mdanalysis/issues/4054)

[YaoYinYing]

Could we (somebody with an ARM mac) build a version or MDAnalysis for ARM macs,

Hi @BradyAJohnston I dig around the cache dir and find the built wheel of MDA and uploaded at here: MDAnalysis-2.2.0-cp310-cp310-macosx_11_0_arm64.whl.zip

The origin file for record sha256 stuff: origin.json.zip

Hope this may be helpful for you.

[YaoYinYing]

I've contacted the MDAnlaysis devs and they are going to try and set up MacOS ARM builds on PyPi for the 2.5.0 release which should hopefully resolve this issue!! (MDAnalysis/mdanalysis#4054)

Wow I can't wait to see it! Then I can close that fix PR full of bullshit!

[YaoYinYing]

Looks like the lack of a target prebuild wheelfile from the official repo may trigger pip to build it from scratch on local machine using gcc/clang(installed via xcode-select).

What I guess is in this circonstance, Python.h will be required as it has been anounced in a specific C/CPP file of MDA source code...

[YaoYinYing]

Wheels can be build using

# build wheel
CPATH=$CPATH:$(python3.10-config --include | sed  -E 's/ ?-I/:/g') ./pip3.10 wheel MDAnalysis==2.4.2 --cache-dir ../cache

# remove downloaded wheels
rm *.whl

# find built MDA wheel
find ../cache/wheels/ -name "*.whl" |grep MDA

[BradyAJohnston]

I still need an M1 to test. @PlethoraChutney would you be up for trying to build a wheel, then installing that in a vanilla blender install?

[PlethoraChutney]

For sure. I just gotta know where to run the above commands!