python-install fails because of stubgen:

[dellaert]

I am in a python3 conda environment, on MacOS 14.6.1,

(py310) (base) macbook-pro-8:build dellaert$ python --version Python 3.10.14

Traceback (most recent call last):
  File "/Users/dellaert/miniforge3/envs/py310/bin/pybind11-stubgen", line 8, in <module>
    sys.exit(main())
  File "/Users/dellaert/miniforge3/envs/py310/lib/python3.10/site-packages/pybind11_stubgen/__init__.py", line 319, in main
    run(
  File "/Users/dellaert/miniforge3/envs/py310/lib/python3.10/site-packages/pybind11_stubgen/__init__.py", line 358, in run
    QualifiedName.from_str(module_name), importlib.import_module(module_name)
  File "/Users/dellaert/miniforge3/envs/py310/lib/python3.10/importlib/__init__.py", line 126, in import_module
Traceback (most recent call last):
    return _bootstrap._gcd_import(name[level:], package, level)  File "/Users/dellaert/miniforge3/envs/py310/bin/pybind11-stubgen", line 8, in <module>

  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    sys.exit(main())
  File "/Users/dellaert/miniforge3/envs/py310/lib/python3.10/site-packages/pybind11_stubgen/__init__.py", line 319, in main
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
    run(
  File "/Users/dellaert/miniforge3/envs/py310/lib/python3.10/site-packages/pybind11_stubgen/__init__.py", line 358, in run
  File "/Users/dellaert/git/github/build/python/gtsam_unstable/__init__.py", line 1, in <module>
    QualifiedName.from_str(module_name), importlib.import_module(module_name)
  File "/Users/dellaert/miniforge3/envs/py310/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
    from .gtsam_unstable import *
ModuleNotFoundError: No module named 'gtsam_unstable.gtsam_unstable'
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/dellaert/git/github/build/python/gtsam/__init__.py", line 9, in <module>
    from gtsam import gtsam, utils
ImportError: cannot import name 'gtsam' from partially initialized module 'gtsam' (most likely due to a circular import) (/Users/dellaert/git/github/build/python/gtsam/__init__.py)
make[3]: *** [python/CMakeFiles/python-unstable-stubs] Error 1
make[3]: *** [python/CMakeFiles/python-stubs] Error 1
make[2]: *** [python/CMakeFiles/python-unstable-stubs.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [python/CMakeFiles/python-stubs.dir/all] Error 2
make[1]: *** [python/CMakeFiles/python-install.dir/rule] Error 2
make: *** [python-install] Error 2

[dellaert]

Actually, even after disabling the stub generation, the library installs fine but the same circular import issue remains:

(py310) (base) macbook-pro-8:github dellaert$ /Users/dellaert/miniforge3/envs/py310/bin/python /Users/dellaert/git/github/python/gtsam/tests/test_Pose3.py
Traceback (most recent call last):
  File "/Users/dellaert/git/github/python/gtsam/tests/test_Pose3.py", line 16, in <module>
    from gtsam.utils.test_case import GtsamTestCase
  File "/Users/dellaert/.local/lib/python3.10/site-packages/gtsam/__init__.py", line 9, in <module>
    from gtsam import gtsam, utils
ImportError: cannot import name 'gtsam' from partially initialized module 'gtsam' (most likely due to a circular import) (/Users/dellaert/.local/lib/python3.10/site-packages/gtsam/__init__.py)

Did something about this change in the last PRs? It's weird as it still works in CI. I remember that this circular thing was an issue a long time ago, but it's weird it pops up again now.

[dellaert]

ls /Users/dellaert/.local/lib/python3.10/site-packages/gtsam
Data                examples            imuBias.py          tests
__init__.py         gtsam.cpython-312-darwin.so noiseModel.py           utils
__pycache__         gtsfm.py            symbol_shorthand.py

Seems the .so file was built for 3.12, but I made sure all cmake versions where 3.10... Does this ring a possible bell, @varunagrawal ?

[varunagrawal]

Check the GTSAM_PYTHON_VERSION and Cmake variables for the Python version in general. This looks like it is being set to different things in different variables.

This issue has bitten me before as well but it's a simple manual change hence I never tried to fix it.

[dellaert]

They are all 3.10

[dellaert]

And, I did:

cmake -DPYTHON_EXECUTABLE=/Users/dellaert/miniforge3/envs/py310/bin/python ..

And it installs in right directory. But somehow something is generating 3.12 binaries?

[dellaert]

It works in a conda 3.12 environment (again with custom cmake)

[ProfFan]

@dellaert I assume you nuked the entire build folder right? Could you paste your cmake log here?

[dellaert]

Did you give it a try on Ubuntu with 3.10?

PS Should CI not test all python versions we support? It should also test that cmake works out of the box in a conda or virtualenv…

[dellaert]

I could do a quick test with a devcontainer…

[ProfFan]

I can test with a Docker in a sec