error using pip to install Opendrop (step 4)

[ranit-mukherjee]

Hi,

I am trying to install Opendrop in my Mac (M1 chip). I have gone through step 4 (installing Boost and Sundials) successfully. But I keep getting this error (full error file attached):

Preparing metadata (pyproject.toml) ... error error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [27 lines of output] Error in sitecustomize; set PYTHONVERBOSE for traceback: AssertionError: Error in sitecustomize; set PYTHONVERBOSE for traceback: AssertionError:

Can you please guide me here? Opendrop error.pdf

[eugenhu]

Hmm, I'm not exactly sure why it's not finding the 'SCons' module. Can you try manually installing scons first, with:

python3 -m pip install SCons

And then try installing opendrop again, this time from the development branch:

python3 -m pip install git+https://github.com/jdber1/opendrop.git@development

[ranit-mukherjee]

Hi Eugen,

I tried the following. The error changed but still fails to install Opendrop. Pasting the error here-

Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... error error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [28 lines of output] Error in sitecustomize; set PYTHONVERBOSE for traceback: AssertionError: Error in sitecustomize; set PYTHONVERBOSE for traceback: AssertionError: /opt/homebrew/Cellar/python@3.10/3.10.6_2/bin/python3.10 -m cython -3 --cplus -o opendrop/features/.checkpoints/colorize.cpp opendrop/features/colorize.pyx Error in sitecustomize; set PYTHONVERBOSE for traceback: AssertionError: /opt/homebrew/Cellar/python@3.10/3.10.6_2/bin/python3.10: No module named cython scons: *** [opendrop/features/.checkpoints/colorize.cpp] Error 1 Traceback (most recent call last): File "/opt/homebrew/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 156, in prepare_metadata_for_build_wheel hook = backend.prepare_metadata_for_build_wheel AttributeError: module 'buildapi' has no attribute 'prepare_metadata_for_build_wheel'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/opt/homebrew/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/opt/homebrew/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/opt/homebrew/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 160, in prepare_metadata_for_build_wheel
      whl_basename = backend.build_wheel(metadata_directory, config_settings)
    File "/private/var/folders/zf/kkc7k0zx10v_68gm9lws56pr0000gn/T/pip-req-build-zmtcb_5i/build_backend/buildapi.py", line 12, in build_wheel
      subprocess.run(['scons', '-Q', 'bdist_wheel', '--build-dir=' + wheel_directory], check=True, env=env)
    File "/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 524, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['scons', '-Q', 'bdist_wheel', '--build-dir=/private/var/folders/zf/kkc7k0zx10v_68gm9lws56pr0000gn/T/pip-modern-metadata-6cgojln9']' returned non-zero exit status 2.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

× Encountered error while generating package metadata. ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details.

[ranit-mukherjee]

I installed the cython module as stated in the above error, but it still failed. The new error message is this:

g++ -o opendrop/fit/younglaplace/.checkpoints/shape.os -c -O3 -std=c++14 -fPIC -Iinclude -I/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10 opendrop/fit/younglaplace/.checkpoints/shape.cpp In file included from opendrop/fit/younglaplace/.checkpoints/shape.cpp:763: include/opendrop/younglaplace.hpp:9:10: fatal error: 'arkode/arkode_erkstep.h' file not found

include <arkode/arkode_erkstep.h>

           ^~~~~~~~~~~~~~~~~~~~~~~~~
  1 error generated.
  scons: *** [opendrop/fit/younglaplace/.checkpoints/shape.os] Error 1
  Traceback (most recent call last):
    File "/opt/homebrew/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 156, in prepare_metadata_for_build_wheel
      hook = backend.prepare_metadata_for_build_wheel
  AttributeError: module 'buildapi' has no attribute 'prepare_metadata_for_build_wheel'

[eugenhu]

Do you know where the sundials headers were installed? E.g. on my linux system, "arkode_erkstep.h" is in "/usr/include/arkode/arkode_erkstep.h". Could yours be in "/usr/local/include/arkode/arkode_erkstep.h"?

[eugenhu]

Possibly, it's in "/opt/homebrew/include/arkode/arkode_erkstep.h". It looks like Homebrew uses the "/opt/homebrew" prefix now for Apple Silicon platforms.

[ranit-mukherjee]

it is in "/opt/homebrew/Cellar/sundials/6.3.0/include/arkode/arkode_erkstep.h".

[eugenhu]

Interesting, could you try running:

CPATH=/opt/homebrew/include python3 -m pip install git+https://github.com/jdber1/opendrop.git@development

And if that fails, try:

CPATH=/opt/homebrew/Cellar/sundials/6.3.0/include python3 -m pip install git+https://github.com/jdber1/opendrop.git@development

CPATH will tell gcc where to look for header files.

[ranit-mukherjee]

Ran both, but failed both of the times. The error message is still similar-

(base) ranit@Ranits-MacBook-Air ~ % C_INCLUDE_PATH=/opt/homebrew/include python3 -m pip install git+https://github.com/jdber1/opendrop.git@development Collecting git+https://github.com/jdber1/opendrop.git@development Cloning https://github.com/jdber1/opendrop.git (to revision development) to /private/var/folders/zf/kkc7k0zx10v_68gm9lws56pr0000gn/T/pip-req-build-lr9er6bu Running command git clone --filter=blob:none --quiet https://github.com/jdber1/opendrop.git /private/var/folders/zf/kkc7k0zx10v_68gm9lws56pr0000gn/T/pip-req-build-lr9er6bu Running command git checkout -b development --track origin/development Switched to a new branch 'development' branch 'development' set up to track 'origin/development'. Resolved https://github.com/jdber1/opendrop.git to commit e18fbf3d7d6f94d02f4846400be518a56b86e73c Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... error error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [43 lines of output] Error in sitecustomize; set PYTHONVERBOSE for traceback: AssertionError: Error in sitecustomize; set PYTHONVERBOSE for traceback: AssertionError: /opt/homebrew/Cellar/python@3.10/3.10.6_2/bin/python3.10 -m cython -3 --cplus -o opendrop/features/.checkpoints/colorize.cpp opendrop/features/colorize.pyx Error in sitecustomize; set PYTHONVERBOSE for traceback: AssertionError: g++ -o opendrop/features/.checkpoints/colorize.os -c -O3 -std=c++14 -fPIC -Iinclude -I/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10 opendrop/features/.checkpoints/colorize.cpp g++ -o opendrop/features/colorize.cpython-310-darwin.so -dynamiclib opendrop/features/.checkpoints/colorize.os -L/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib -lpython3.10 /opt/homebrew/Cellar/python@3.10/3.10.6_2/bin/python3.10 -m cython -3 --cplus -o opendrop/fit/needle/.checkpoints/hough.cpp opendrop/fit/needle/hough.pyx Error in sitecustomize; set PYTHONVERBOSE for traceback: AssertionError: g++ -o opendrop/fit/needle/.checkpoints/hough.os -c -O3 -std=c++14 -fPIC -Iinclude -I/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10 opendrop/fit/needle/.checkpoints/hough.cpp g++ -o opendrop/fit/needle/hough.cpython-310-darwin.so -dynamiclib opendrop/fit/needle/.checkpoints/hough.os -L/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib -lpython3.10 -lm /opt/homebrew/Cellar/python@3.10/3.10.6_2/bin/python3.10 -m cython -3 --cplus -o opendrop/fit/younglaplace/.checkpoints/shape.cpp opendrop/fit/younglaplace/shape.pyx Error in sitecustomize; set PYTHONVERBOSE for traceback: AssertionError: g++ -o opendrop/fit/younglaplace/.checkpoints/shape.os -c -O3 -std=c++14 -fPIC -Iinclude -I/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10 opendrop/fit/younglaplace/.checkpoints/shape.cpp In file included from opendrop/fit/younglaplace/.checkpoints/shape.cpp:763: include/opendrop/younglaplace.hpp:9:10: fatal error: 'arkode/arkode_erkstep.h' file not found

include <arkode/arkode_erkstep.h>

[eugenhu]

I see, can you try using CPATH instead of C_INCLUDE_PATH. I've edited my previous comment, I think C_INCLUDE_PATH is for C only and not C++, apologies.

[ranit-mukherjee]

Thank you for your help! I changed it to CPATH. But the same problem of arkode_erkstep not being found still persists.

[eugenhu]

I've patched the build script to pass "/opt/homebrew/include" directly to gcc as a command line argument. Can you try installing from the new branch:

python3 -m pip install git+https://github.com/jdber1/opendrop.git@issue43

[ranit-mukherjee]

This solved the previous problem, but looks like can not access the sundials_arkode library:

g++ -o opendrop/fit/younglaplace/shape.cpython-310-darwin.so -dynamiclib opendrop/fit/younglaplace/.checkpoints/shape.os -L/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib -lpython3.10 -lsundials_arkode -lsundials_nvecserial ld: library not found for -lsundials_arkode clang: error: linker command failed with exit code 1 (use -v to see invocation)

[eugenhu]

Apologies for the late reply, I think the best way to compile these C extension modules (in macOS) that depend on other shared libs is to hardcode "/opt/homebrew/lib" into the binaries' rpath. I've added another patch to the issue43 branch, could you try installing again?:

python3 -m pip install git+https://github.com/jdber1/opendrop.git@issue43

[ranit-mukherjee]

No worries! You have been super helpful. I am very new to python, so still learning the basics.

I tried the new patch, it shows the same problem of not finding sundials_arkode library

[eugenhu]

Sorry again for the back and forth. It might take a few more trys and errors before we reach a successful build, so thanks for your patience. I've made another patch, when you're free can you try installing again:

python3 -m pip install git+https://github.com/jdber1/opendrop.git@issue43

Hopefully the linker is able to find the libraries and finish compiling; but it may not be able to run successfully yet since one might need to add "/opt/homebrew/lib" to Homebrew's python executable's rpaths. If the compilation succeeds but running the app (with python -m opendrop) fails, try running with the command DYLD_LIBRARY_PATH=/opt/homebrew/lib python -m opendrop instead.

I will be away for the next 2.5 weeks so my further replies may be delayed until then.

[ranit-mukherjee]

This one compiled perfectly! Thank you so much. But running the app is now crashing python with the following error (Screenshot). I tried adding the LIBRARY_PATH command also, but similar error.

[eugenhu]

Thanks for your patience again. Can you try running

DYLD_PRINT_LIBRARIES=1 python -m opendrop

This will fail, but it should print some diagnostic information into the terminal, could you copy and paste that here?

I don't have an easy solution in mind to get this working on Homebrew on Apple Silicon. I'd have to go through a lot more trial-and-error and I don't want to burden you with that if running opendrop on your M1 mac is not super important to you. There are pre-built windows binaries you can try if you need to use opendrop and have access to a windows machine. Thanks for the cooperation so far though, you've been very helpful. But I'm happy to leave this issue open for now until someone else with an apple silicon mac is able to troubleshoot this.

[ranit-mukherjee]

Hi eugenhu,

Sorry for the late comment. Thanks again for all the help. Even though I could not make it work, I got to learn some stuff from your comments. I ran the command you mentioned. It showed the same "zsh segmentation fault" error with the python stopped working window. I Don't think that is what you were hoping for. Anyway, hopefully I can figure something out in the windows system.