search

omnetpp / omnetpp

OMNeT++ Discrete Event Simulator
https://omnetpp.org
588 stars 150 forks source link

Problems Linking with Conda Python #1236

Open gatsby0203 opened 4 days ago

gatsby0203 commented 4 days ago

Platform

Describe the bug OMNeT++ is currently using python3-config --ldflags --embed to generate Python linking flags, but this approach causes issues in environments like Conda. Specifically, Conda’s Python is built with Py_ENABLE_SHARED=0 by default, yet still generates libpython. As described in https://github.com/conda/conda-build/issues/2738

python3-config is used for embedding a Python interpreter in another program, not for getting the flags to use for building extension modules. For that you should use:

python -c "import sysconfig; print(sysconfig.get_config_vars('LDSHARED'))"

To resolve the issue, I replaced the following lines in OMNeT++’s configure script https://github.com/omnetpp/omnetpp/blob/23482d43991ccec9858a588141f1dc0a080a1252/configure#L5610-L5612 with

  PYTHON_LDFLAGS=$(python3 -c "import sysconfig; print(' '.join(sysconfig.get_config_vars('LDFLAGS')))")
  PYTHON_LDFLAGS="$PYTHON_LDFLAGS $($PYTHON_CONFIG --libs --embed)"

It works.

This is only a temporary workaround that I applied to get the build working. I hope the developers can provide an official solution or guidance on how to better handle Python linking in Conda and similar environments.

rhornig commented 4 days ago

While currently it is not needed, we are using the --embed flag because we plan to introduce more python support in future versions of omnetpp (master branch already contains code that embeds the interpreter) so we are preparing the build process for that. It is good to know that in conda this is problematic.

I'm missing what is the actual issue (is this a build failure?). Would not removing the --embed flag be enough as a workaround?

Generally, we encourage users to use a virtual python environemnt for omnet and omnet can set up its own environment in the omnetpp-6.0/.venv folder. If you use the ./install.sh script, it will be set up automatically.

gatsby0203 commented 4 days ago

While currently it is not needed, we are using the --embed flag because we plan to introduce more python support in future versions of omnetpp (master branch already contains code that embeds the interpreter) so we are preparing the build process for that. It is good to know that in conda this is problematic.

I'm missing what is the actual issue (is this a build failure?). Would not removing the --embed flag be enough as a workaround?

Generally, we encourage users to use a virtual python environemnt for omnet and omnet can set up its own environment in the omnetpp-6.0/.venv folder. If you use the ./install.sh script, it will be set up automatically.

Thank you for your response. The actual issue I encountered is that without the modification, the configure script fails to run and outputs the following error:

checking for python3-dev with CFLAGS="-I/opt/homebrew/Caskroom/miniconda/base/envs/simEnv/include/python3.10 -I/opt/homebrew/Caskroom/miniconda/base/envs/simEnv/include/python3.10" LIBS="-L/opt/homebrew/Caskroom/miniconda/base/envs/simEnv/lib/python3.10/config-3.10-darwin -lpython3.10 -ldl -framework CoreFoundation"... no
configure: error: Python support was configured (WITH_SCAVE_PYTHON_BINDINGS=yes), but the Python 3 development package was not found.

I believe the root cause is that in a Conda environment, Py_ENABLE_SHARED=0 by default. This causes python-config to return the following library search path:

-L/opt/homebrew/Caskroom/miniconda/base/envs/simEnv/lib/python3.10/config-3.10-darwin

This path does not include libpython, which is required for linking.

The --embed flag ensures that -lpython is included in the linking flags. Configure script still fails with --embed removed, the test code needs link to libpython to compile.

configure:5705: clang++ -o conftest -I/opt/homebrew/Caskroom/miniconda/base/envs/simEnv/include/python3.10 -I/opt/homebrew/Caskroom/miniconda/base/envs/simEnv/include/python3.10 -I/opt/homebrew/opt/make/include -L/opt/homebrew/opt/make/lib -Wl,-rpath,$(OMNETPP_LIB_DIR) -Wl,-rpath,. conftest.cpp -L/opt/homebrew/Caskroom/miniconda/base/envs/simEnv/lib/python3.10/config-3.10-darwin -ldl -framework CoreFoundation >&5
ld: warning: search path '/opt/homebrew/opt/make/lib' not found
Undefined symbols for architecture arm64:
  "_Py_Initialize", referenced from:
      _main in conftest-5966ae.o
ld: symbol(s) not found for architecture arm64
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

I also tried using the install.sh script, which is very helpful in setting up the environment automatically. However, this issue is specific to Conda’s Python setup with Py_ENABLE_SHARED=0, which causes issues with linking libpython during the configuration step. Conda is commonly used for managing multiple Python versions and environments, I hope this issue can be properly addressed.

rhornig commented 1 day ago

Thanks for the details. At the moment, in 6.1, python-dev is used ONLY to create native python bindings to read in sca and vec files in the omneptt.scave module IF you use that module from the command line (i.e. if you write some python code that processes data and runs outside of the IDE. In this sense, the testing code in the configure script is a bit too demanding and unnecessarily tests for the python library.

But no matter what, in newer versions of OMNeT++ we will use python embedding. If miniconda does not provide that then it can't be used with OMNeT++ as a dependency.

Is it a problem that OMNET uses a normal (non-conda) version of python? The install.sh script already installs python@3 with brew. Maybe we should force omnet to use the default python@3 package even if miniconda is already installed?

Anyway, thanks for pointing this out. I will experiment with this scenario.

rhornig commented 1 day ago

As a workaround you can:

gatsby0203 commented 1 day ago

Thanks for the details. At the moment, in 6.1, python-dev is used ONLY to create native python bindings to read in sca and vec files in the omneptt.scave module IF you use that module from the command line (i.e. if you write some python code that processes data and runs outside of the IDE. In this sense, the testing code in the configure script is a bit too demanding and unnecessarily tests for the python library.

But no matter what, in newer versions of OMNeT++ we will use python embedding. If miniconda does not provide that then it can't be used with OMNeT++ as a dependency.

Is it a problem that OMNET uses a normal (non-conda) version of python? The install.sh script already installs python@3 with brew. Maybe we should force omnet to use the default python@3 package even if miniconda is already installed?

Anyway, thanks for pointing this out. I will experiment with this scenario.

Interestingly, I noticed that the Python installed via Homebrew also has Py_ENABLE_SHARED=0, but it works fine because libpython is already compiled and available in the config-3.x-darwin directory. It seems they intend for libpython in the lib directory not to be used for embedding purposes.

It requires pre-compiling libpython using the scripts found in the config-3.x-darwin directory for binding Conda Python. (I’ve tried linking the libpython from the Conda Python'slib directory, and it works for now, though I’m unsure if there will be any side effects.)

Anyway, I’ve managed to get everything working on my end, so I think we can close this issue.

Thanks again for your help!