Python modules not installed despite the --python option

[diminDDL]

I am trying to install opemEMS while following the installation instructions to include the python interface. After installing dependencies and starting the build I get the following output:

> ./update_openEMS.sh ~/opt/openEMS --python
enabling Python Extension build
setting install path to: /home/dmytro/opt/openEMS
logging build output to: /home/dmytro/Documents/projects/openEMS-Project/build_20240403_004127.log
running cmake in tmp dir: /tmp/tmp.UMYLr24AKr
CMake Warning (dev) at CMakeLists.txt:5 (PROJECT):
  cmake_minimum_required() should be called prior to this top-level project()
  call.  Please see the cmake-commands(7) manual for usage documentation of
  both commands.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Deprecation Warning at CMakeLists.txt:7 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

build openEMS and dependencies ... please wait
build successful, cleaning up tmp dir ...
build CSXCAD python module ... please wait
build openEMS python module ... please wait
 -------- 
openEMS and all modules have been updated successfully...

% add the required paths to Octave/Matlab:
addpath('/home/dmytro/opt/openEMS/share/openEMS/matlab')
addpath('/home/dmytro/opt/openEMS/share/CSXCAD/matlab')

% optional additional pckages:

Have fun using openEMS

log file: build_20240403_004127.log

But when I look into /home/dmytro/opt/openEMS/share/openEMS/ and /home/dmytro/opt/openEMS/share/CSXCAD/ I only see the matlab folder. From my understanding there should also be a python folder there, but there is none. Trying to import openems in python REPL also fails.

What am I doing wrong, any help would be greatly appreciated. Thank you.

[LubomirJagos42]

Hi, I encountered this problem since I was playing with openEMS compile at RHEL (RadHat Enterprise Linux). It says there: CMake Deprecation Warning at CMakeLists.txt:7 (cmake_minimum_required): Compatibility with CMake < 3.5 will be removed from a future version of CMake.

The problem is that you have old version of cmake, you need at least 3.5 version. Also then openEMS build script must be updated to use cmake3 command, here is what I did: sudo yum install -y cmake3.x86_64 sed -i 's/cmake -DBUILD/cmake3 -DBUILD/g' update_openEMS.sh

First command will install cmake3 version, that one is for RedHat linux but I bet ubuntu or other will have similar like: sudo install cmake3

Second command will open script update_openEMS.sh and edit line: cmake -DBUILD to: cmake3 -DBUILD which will use cmake 3 to make script.

Attaching to this comment my build script which is installing all needed packages and downloading old version of openEMS and compile it from source, it's doing some edit stuff as I tried to install it to 10years old RHEL virtual machine therefore cannot used new libraries and have to look for old versions of boost, CGAL lib and so, but this script could be some inspiration what must be done to compile old version of openEMS.

As result you should have in ~/opt/openEMS/ directories for matlab and python if you enable it and you should be able to run openEMS itself.

openEMS_install_complete.txt

[lasofeli]

The problem is that you have old version of cmake, you need at least 3.5 version. Also then openEMS build script must be updated to use cmake3 command, here is what I did:

Doesn't the deprecation warning refer to the cmake version openEMS is using and not the version of Cmake that diminDDL had?

CMake Deprecation Warning at CMakeLists.txt:7 (cmake_minimum_required):

[diminDDL]

Sorry for the late reply, but yeah I assumed that the deprecation warning is about the cmake version of the stuff I am building and not cmake itself, since I build a lot of other stuff on the daily and even the projects with these warning compile generally fine.

[edit] miss clicked the close button.

[lasofeli]

Manually building the packages yourself still works. You may need to run the make and make install commands as root. I had to copy the generated files from the install directory to /usr for gcc to detect them. Last thing I'd note is that, if you plan on running the python setup script in a venv, you may need to remove the --user to get it to install in the venv.

I did this on Debian 12.

[keixthb]

@lasofeli @thliebig @aWZHY0yQH81uOYvH I think this is similar to this issue found here. How did you get the openEMS-Project/openEMS/python/setup.py to run? I get an error with cython not being able to find the header files (centos 9 stream x86_64).

Steps to reproduce

I have to build a lot of the dependencies manually in centos, but my general process is this:

git clone --recurse-submodules --shallow-submodules --depth=10 https://github.com/thliebig/openEMS-Project ~/openems

..and cd into the repo, making a build directory:

cd ~/openems/openEMS-Project; mkdir build

In the process of fetching the dependencies, I run:

python3 -m venv my_openems_env; source my_openems_env/bin/activate; pip install --upgrade pip

which gives me this:

└── openems
    └── openEMS-Project
        ├── AppCSXCAD
        ├── build
        ├── CMakeLists.txt
        ├── CSXCAD
        ├── CTB
        ├── doc-src
        ├── fparser
        ├── hyp2mat
        ├── make_tarball.sh
        ├── my_openems_env
        ├── openEMS
        ├── openEMS.rb
        ├── other
        ├── QCSXCAD
        ├── readme.md
        └── update_openEMS.sh

Then, after getting all the dependencies, I run:

./update_openEMS.sh ~/openems/openEMS-Project/build --with-hyp2mat --with-CTB --python

I'm able to get everything c++ built just fine (I can write up some updated centos instructions if anyone would like), but for some reason, the python bindings don't build for me--and I am not very experienced with setup.py.

Error Message

The log file (build_20240527_150646.log)says that:

Error message

running install
Checking .pth file support in /home/swbova/.local/lib/python3.9/site-packages/
/home/swbova/openems/openEMS-Project/ems_env/bin/python3 -E -c pass
TEST FAILED: /home/swbova/.local/lib/python3.9/site-packages/ does NOT support .pth files
bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /home/swbova/.local/lib/python3.9/site-packages/

and your PYTHONPATH environment variable currently contains:

    ''

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:

  https://setuptools.readthedocs.io/en/latest/easy_install.html#custom-installation-locations

...which leads me to try to run the python install manually. I got the openEMS-Project/CSXCAD/python/setup.py installed no issues using:

python setup.py build_ext -I~/openems/openEMS-Project/openEMS/include -L~/openems/openEMS-Project/openEMS/lib -R~/openems/openEMS-Project/openEMS/lib --user

...but when i run the same command in openems/openEMS-Project/openEMS/python/, I get the following error:

Error message

running build_ext
building 'openEMS.openEMS' extension
gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/home/swbova/openems/openEMS-Project/CSXCAD/ -I~/openems/openEMS-Project/openEMS/include -I/home/swbova/openems/openEMS-Project/ems_env/include -I/usr/include/python3.9 -c /home/swbova/openems/openEMS-Project/openEMS/python/openEMS/openEMS.cpp -o build/temp.linux-x86_64-3.9/home/swbova/openems/openEMS-Project/openEMS/python/openEMS/openEMS.o
/home/swbova/openems/openEMS-Project/openEMS/python/openEMS/openEMS.cpp:1263:10: fatal error: CSXCAD/ParameterObjects.h: No such file or directory
 1263 | #include "CSXCAD/ParameterObjects.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

Does anyone know where the openEMS/python/setup.py looks for its files? Or know how to get the setup to run? Thanks in advance.

[aWZHY0yQH81uOYvH]

@keixthb Try running setup.py with -I, -L, and -R pointing to the final install directory. The build process for openEMS needs CSXCAD, and so when you provide ~/openems/openEMS-Project/openEMS/include (which doesn't exist), it can't find the CSXCAD headers. If you point it to the final install directory after all the C++ stuff has been installed, all the required files should be there. This is what I did in the Homebrew build script.

[keixthb]

@aWZHY0yQH81uOYvH An oversight on my part, i see the files that the setup.py is looking for here:

[swbova@voronoi1 CSXCAD]$ pwd
/home/swbova/openems/openEMS-Project/build/include/CSXCAD
[swbova@voronoi1 CSXCAD]$ ls
ContinuousStructure.h   CSPrimCylinder.h          CSPrimPoint.h             CSPrimSphere.h           CSPropDebyeMaterial.h       CSPropExcitation.h       CSPropProbeBox.h  CSUseful.h
CSBackgroundMaterial.h  CSPrimCylindricalShell.h  CSPrimPolygon.h           CSPrimSphericalShell.h   CSPropDiscMaterial.h        CSPropLorentzMaterial.h  CSPropResBox.h    CSXCAD_Global.h
CSFunctionParser.h      CSPrimitives.h            CSPrimPolyhedron.h        CSPrimUserDefined.h      CSPropDispersiveMaterial.h  CSPropLumpedElement.h    CSPropUnknown.h   ParameterCoord.h
CSPrimBox.h             CSPrimLinPoly.h           CSPrimPolyhedronReader.h  CSPrimWire.h             CSPropDumpBox.h             CSPropMaterial.h         CSRectGrid.h      ParameterObjects.h
CSPrimCurve.h           CSPrimMultiBox.h          CSPrimRotPoly.h           CSPropConductingSheet.h  CSProperties.h              CSPropMetal.h            CSTransform.h

and the lib is here:

[swbova@voronoi1 lib]$ pwd
/home/swbova/openems/openEMS-Project/build/lib
[swbova@voronoi1 lib]$ ls
libCSXCAD.so    libCSXCAD.so.0.6.3  libfparser.so.4      libnf2ff.so    libnf2ff.so.0.1.0  libopenEMS.so.0       libQCSXCAD.so    libQCSXCAD.so.0.6.3
libCSXCAD.so.0  libfparser.so       libfparser.so.4.5.1  libnf2ff.so.0  libopenEMS.so      libopenEMS.so.0.0.36  libQCSXCAD.so.0
[swbova@voronoi1 lib]$

But when I do this, the result is the same?

(ems_env) [swbova@voronoi1 python]$ python setup.py build_ext -I /home/swbova/openems/openEMS-Project/build/include/CSXCAD/ -L /home/swbova/openems/openEMS-Project/build/lib/ -R /home/swbova/openems/openEMS-Project/build/lib/
running build_ext
building 'openEMS.openEMS' extension
gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/home/swbova/openems/openEMS-Project/CSXCAD/ -I/home/swbova/openems/openEMS-Project/build/include/CSXCAD/ -I/home/swbova/openems/openEMS-Project/ems_env/include -I/usr/include/python3.9 -c /home/swbova/openems/openEMS-Project/openEMS/python/openEMS/openEMS.cpp -o build/temp.linux-x86_64-3.9/home/swbova/openems/openEMS-Project/openEMS/python/openEMS/openEMS.o
/home/swbova/openems/openEMS-Project/openEMS/python/openEMS/openEMS.cpp:1263:10: fatal error: CSXCAD/ParameterObjects.h: No such file or directory
 1263 | #include "CSXCAD/ParameterObjects.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

For reference, this is my ~/openems/openEMS-Project/openEMS/python/setup.py

setup.py

# -*- coding: utf-8 -*-
"""
Created on Sun Dec 13 23:48:22 2015

@author: thorsten
"""

from setuptools import setup
from setuptools import Extension
from Cython.Build import cythonize

import os, sys
ROOT_DIR = os.path.dirname(__file__)

sys.path.append(os.path.join(ROOT_DIR,'..','..','CSXCAD','python'))

extensions = [
    Extension("*", [os.path.join(os.path.dirname(__file__), "openEMS","*.pyx")],
        language="c++",             # generate C++ code
        libraries    = ['CSXCAD','openEMS', 'nf2ff']),
]

setup(
  name="openEMS",
  version = '0.0.36',
  description = "Python interface for the openEMS FDTD library",
  classifiers = [
      'Development Status :: 3 - Alpha',
      'Intended Audience :: Developers',
      'Intended Audience :: Information Technology',
      'Intended Audience :: Science/Research',
      'License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)',
      'Programming Language :: Python',
      'Programming Language :: Python :: 3',
      'Programming Language :: Python :: 3.9',
      'Programming Language :: Python :: 3.10',
      'Programming Language :: Python :: 3.11',
      'Programming Language :: Python :: 3.12',
      'Programming Language :: Python :: Implementation :: CPython',
      'Topic :: Scientific/Engineering',
      'Topic :: Software Development :: Libraries :: Python Modules',
      'Operating System :: POSIX :: Linux',
      'Operating System :: Microsoft :: Windows',
  ],
  author = 'Thorsten Liebig',
  author_email = 'Thorsten.Liebig@gmx.de',
  maintainer = 'Thorsten Liebig',
  maintainer_email = 'Thorsten.Liebig@gmx.de',
  url = 'https://openEMS.de',
  packages=["openEMS", ],
  package_data={'openEMS': ['*.pxd']},
  python_requires='>=3.9',
  install_requires=[
    'cython>=3.0.6',  # Apache License 2.0 (https://github.com/cython/cython/blob/master/LICENSE.txt)
    'h5py>=3.10.0',   # BSD 3-Clause (https://github.com/h5py/h5py/blob/master/LICENSE)
    'numpy>=1.26.2',  # BSD 3-Clause (https://github.com/numpy/numpy/blob/main/LICENSE.txt)
  ],
  ext_modules = cythonize(extensions, language_level = 3)
 )

[aWZHY0yQH81uOYvH]

@keixthb You want something like

python setup.py build_ext -I /home/swbova/openems/openEMS-Project/build/include -L /home/swbova/openems/openEMS-Project/build/lib -R /home/swbova/openems/openEMS-Project/build/lib

Notice it's looking for a file called CSXCAD/ParameterObjects.h, which includes the CSXCAD directory, so -I can't have /CSXCAD at the end, which would result in it trying to find a file build/include/CSXCAD/CSXCAD/ParameterObjects.h.

Also, before you asked about where it's looking by default. The gcc command it's running has the following -I arguments passed to it:

-I/home/swbova/openems/openEMS-Project/CSXCAD/
-I/home/swbova/openems/openEMS-Project/build/include/CSXCAD/
-I/home/swbova/openems/openEMS-Project/ems_env/include
-I/usr/include/python3.9

[keixthb]

@aWZHY0yQH81uOYvH This is the solution right here. The python build worked for me, no issues when I used these paths. Thank you so much!

[dan-gies]

For anyone else that comes across this - I'm fumbling my way through the build (still having the issue where using --python doesn't work out-of-the-box) and this PR helped a lot:

https://github.com/thliebig/openEMS/pull/150/commits/af85ccb44b792598d4b734d116384533fad1655c