[Enhancement Request] Allow importing Python package with C-extension e.g. numpy

wta12 commented 1 year ago

Enhancement Request

Allow importing package with C-extension e.g. numpy

Detailed Description

OS SDK 3.5.1 returns error if execute python script with numpy as dependencies "OpenStudio/OpenStudio_3_5_1/bin/openstudio labs --verbose --python_home /usr/local/anaconda3/envs/pythonMeasure --python_path /usr/local/anaconda3/envs/pythonMeasure/lib/python3.8/site-packages execute_python_script python_script_test.py" Return error:

Error message: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

Possible Implementation:

EP instance in same OS SDK installation allows Plugin:Instance (pythons script) with numpy imported

tijcolem commented 1 year ago

@Myoldmopar I think this one is related to how E+ manages python and python packages?

tijcolem commented 1 year ago

@wta12 Also, have you tried setting the PYTHONPATH env to point to the pip package repo and then run it?

e.g. export PYTHONPATH=/path/to/my/modules/

Myoldmopar commented 1 year ago

I'm happy to help, but I'd need to dive into the way OpenStudio packages and uses EnergyPlus. There is certainly a method for this to work with raw EnergyPlus, although it's admittedly not the most elegant solution. I suspect @jmarrec would have more to offer in the near term, if available.

jmarrec commented 1 year ago

I have it fixed locally. But I'd like to discuss with @kbenne about it, because I had to change the Unix (well, Ubuntu to be exact, I think it works on mac...) dlopen flags from RTLD_LAZY | RTLD_LOCAL to RTLD_GLOBAL | RTLD_NOW when using python.

On unix, the C native ext are NOT linked to libpython.so. Our libpythonengine.so IS linked to it. If we use RTLD_LOCAL, we do not export these libpython.so symbols and as such we get unresolved symbols at runtime, such as undefined symbol: PyFloat_Type. The alternative is to find a clever way also dlopen the libpython.so, this one with the RTLD_GLOBAL visibility.

https://stackoverflow.com/questions/67891197/ctypes-cpython-39-x86-64-linux-gnu-so-undefined-symbol-pyfloat-type-in-embedd

Currently

Mac

$ openstudio labs --python_path /Users/julien/.pyenv/versions/3.8.12/lib/python3.8/site-packages --python_path /Users/julien/.pyenv/versions/3.8.12/lib/python3.8/lib-dynload --pyexecute "import numpy as np; print(np.array([1, 2, 3]))"
┌────────────────────────────────────────────────────────────────────────────────┐
│         The `labs` command is experimental - Do not use in production          │
└────────────────────────────────────────────────────────────────────────────────┘
[1 2 3]

Ubuntu 20.04

$ openstudio labs --python_path /home/julien/.pyenv/versions/3.8.13/lib/python3.8/site-packages --python_path /home/julien/.pyenv/versions/3.8.13/lib/python3.8/lib-dynload --pyexecute "import numpy as np; print(np.array([1, 2, 3]))"
┌────────────────────────────────────────────────────────────────────────────────┐
│         The `labs` command is experimental - Do not use in production          │
└────────────────────────────────────────────────────────────────────────────────┘
Traceback (most recent call last):
  File "/home/julien/.pyenv/versions/3.8.13/lib/python3.8/site-packages/numpy/core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/home/julien/.pyenv/versions/3.8.13/lib/python3.8/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/home/julien/.pyenv/versions/3.8.13/lib/python3.8/site-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: /home/julien/.pyenv/versions/3.8.13/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyObject_SelfIter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/julien/.pyenv/versions/3.8.13/lib/python3.8/site-packages/numpy/__init__.py", line 141, in <module>
    from . import core
  File "/home/julien/.pyenv/versions/3.8.13/lib/python3.8/site-packages/numpy/core/__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.8 from "/usr/bin/python3"
  * The NumPy version is: "1.24.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: /home/julien/.pyenv/versions/3.8.13/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyObject_SelfIter

terminate called after throwing an instance of 'std::runtime_error'
  what():  Error executing Python code
Aborted (core dumped)

NREL / OpenStudio