pybind11 missing header inside conda

[lgarrison]

The pybind11 compilation fails because the pybind11.h include path is missing from the compiler invocation when installing inside a conda environment (first noticed this while debugging #7). When installing inside a venv, there doesn't seem to be any problem. The error message is:

(nufft-ls-py39) lgarrison@ccalin008:~/nufft-ls$ pip install -e .
[...]
src/nufft_ls/cpu.cpp:1:31: fatal error: pybind11/pybind11.h: No such file or directory
 #include <pybind11/pybind11.h>
                               ^
compilation terminated.
error: command '/usr/lib64/ccache/gcc' failed with exit code 1

Full details:

```console lgarrison@ccalin008:~/nufft-ls$ rm -rf build/ src/nufft_ls.egg-info/ src/nufft_ls/*.so lgarrison@ccalin008:~/nufft-ls$ conda create -n nufft-ls-py39 'python<3.10' Collecting package metadata (current_repodata.json): done Solving environment: done ## Package Plan ## environment location: /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39 added / updated specs: - python[version='<3.10'] The following NEW packages will be INSTALLED: _libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge _openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-1_gnu bzip2 conda-forge/linux-64::bzip2-1.0.8-h7f98852_4 ca-certificates conda-forge/linux-64::ca-certificates-2021.10.8-ha878542_0 ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.36.1-hea4e1c9_2 libffi conda-forge/linux-64::libffi-3.4.2-h7f98852_5 libgcc-ng conda-forge/linux-64::libgcc-ng-11.2.0-h1d223b6_12 libgomp conda-forge/linux-64::libgomp-11.2.0-h1d223b6_12 libnsl conda-forge/linux-64::libnsl-2.0.0-h7f98852_0 libuuid conda-forge/linux-64::libuuid-2.32.1-h7f98852_1000 libzlib conda-forge/linux-64::libzlib-1.2.11-h36c2ea0_1013 ncurses conda-forge/linux-64::ncurses-6.3-h9c3ff4c_0 openssl conda-forge/linux-64::openssl-3.0.0-h7f98852_2 pip conda-forge/noarch::pip-22.0.3-pyhd8ed1ab_0 python conda-forge/linux-64::python-3.9.10-hc74c709_2_cpython python_abi conda-forge/linux-64::python_abi-3.9-2_cp39 readline conda-forge/linux-64::readline-8.1-h46c0cb4_0 setuptools conda-forge/linux-64::setuptools-60.8.1-py39hf3d152e_0 sqlite conda-forge/linux-64::sqlite-3.37.0-h9cd32fc_0 tk conda-forge/linux-64::tk-8.6.11-h27826a3_1 tzdata conda-forge/noarch::tzdata-2021e-he74cb21_0 wheel conda-forge/noarch::wheel-0.37.1-pyhd8ed1ab_0 xz conda-forge/linux-64::xz-5.2.5-h516909a_1 zlib conda-forge/linux-64::zlib-1.2.11-h36c2ea0_1013 Proceed ([y]/n)? y Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate nufft-ls-py39 # # To deactivate an active environment, use # # $ conda deactivate lgarrison@ccalin008:~/nufft-ls$ conda activate nufft-ls-py39 (nufft-ls-py39) lgarrison@ccalin008:~/nufft-ls$ which python ~/miniconda3/envs/nufft-ls-py39/bin/python (nufft-ls-py39) lgarrison@ccalin008:~/nufft-ls$ python -m pip install -U pip python -Requirement already satisfied: pip in /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/lib/python3.9/site-packages (22.0.3) m (nufft-ls-py39) lgarrison@ccalin008:~/nufft-ls$ python -m pip install -e . -v Using pip 22.0.3 from /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/lib/python3.9/site-packages/pip (python 3.9) Obtaining file:///mnt/home/lgarrison/nufft-ls Running command pip subprocess to install build dependencies Collecting setuptools<60.6,>=42 Using cached setuptools-60.5.0-py3-none-any.whl (958 kB) Collecting wheel Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) Collecting pybind11~=2.9 Using cached pybind11-2.9.1-py2.py3-none-any.whl (211 kB) Installing collected packages: wheel, setuptools, pybind11 Successfully installed pybind11-2.9.1 setuptools-60.5.0 wheel-0.37.1 Installing build dependencies ... done Running command Checking if build backend supports build_editable Checking if build backend supports build_editable ... done Running command Getting requirements to build wheel running egg_info creating src/nufft_ls.egg-info writing manifest file 'src/nufft_ls.egg-info/SOURCES.txt' writing manifest file 'src/nufft_ls.egg-info/SOURCES.txt' Getting requirements to build wheel ... done Running command Preparing metadata (pyproject.toml) running dist_info creating /tmp/pip-modern-metadata-6lxhv1u5/nufft_ls.egg-info writing manifest file '/tmp/pip-modern-metadata-6lxhv1u5/nufft_ls.egg-info/SOURCES.txt' writing manifest file '/tmp/pip-modern-metadata-6lxhv1u5/nufft_ls.egg-info/SOURCES.txt' Preparing metadata (pyproject.toml) ... done Collecting numpy Using cached numpy-1.22.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB) Collecting astropy Using cached astropy-5.0.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (11.1 MB) Collecting threadpoolctl Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB) Collecting PyYAML>=3.13 Using cached PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (661 kB) Collecting pyerfa>=2.0 Using cached pyerfa-2.0.0.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (742 kB) Collecting packaging>=19.0 Using cached packaging-21.3-py3-none-any.whl (40 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Using cached pyparsing-3.0.7-py3-none-any.whl (98 kB) Installing collected packages: threadpoolctl, PyYAML, pyparsing, numpy, pyerfa, packaging, astropy, nufft-ls changing mode of /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/bin/f2py to 775 changing mode of /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/bin/f2py3 to 775 changing mode of /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/bin/f2py3.9 to 775 changing mode of /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/bin/fits2bitmap to 775 changing mode of /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/bin/fitscheck to 775 changing mode of /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/bin/fitsdiff to 775 changing mode of /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/bin/fitsheader to 775 changing mode of /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/bin/fitsinfo to 775 changing mode of /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/bin/samp_hub to 775 changing mode of /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/bin/showtable to 775 changing mode of /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/bin/volint to 775 changing mode of /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/bin/wcslint to 775 Running setup.py develop for nufft-ls Running command python setup.py develop running develop /tmp/pip-build-env-b_r8zry7/overlay/lib/python3.9/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools. warnings.warn( /tmp/pip-build-env-b_r8zry7/overlay/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running egg_info writing manifest file 'src/nufft_ls.egg-info/SOURCES.txt' running build_ext gcc -pthread -B /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -I/mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include/python3.9 -c flagcheck.cpp -o flagcheck.o -march=native gcc -pthread -B /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -I/mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include/python3.9 -c flagcheck.cpp -o flagcheck.o -Ofast gcc -pthread -B /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -I/mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include/python3.9 -c flagcheck.cpp -o flagcheck.o -fopenmp gcc -pthread -B /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -I/mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include/python3.9 -c flagcheck.cpp -o flagcheck.o -funroll-loops gcc -pthread -B /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -I/mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include/python3.9 -c flagcheck.cpp -o flagcheck.o -ffast-math gcc -pthread -B /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -I/mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include/python3.9 -c flagcheck.cpp -o flagcheck.o -std=c++17 gcc: error: unrecognized command line option ‘-std=c++17’ gcc -pthread -B /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -I/mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include/python3.9 -c flagcheck.cpp -o flagcheck.o -std=c++14 gcc: error: unrecognized command line option ‘-std=c++14’ gcc -pthread -B /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -I/mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include/python3.9 -c flagcheck.cpp -o flagcheck.o -std=c++11 creating build creating build/temp.linux-x86_64-3.9 creating build/temp.linux-x86_64-3.9/src creating build/temp.linux-x86_64-3.9/src/nufft_ls gcc -pthread -B /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -O2 -isystem /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include -fPIC -Iinclude/ -I/mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/include/python3.9 -c src/nufft_ls/cpu.cpp -o build/temp.linux-x86_64-3.9/src/nufft_ls/cpu.o -std=c++11 -fvisibility=hidden -g0 -march=native -Ofast -fopenmp -funroll-loops -ffast-math src/nufft_ls/cpu.cpp:1:31: fatal error: pybind11/pybind11.h: No such file or directory #include ^ compilation terminated. error: command '/usr/lib64/ccache/gcc' failed with exit code 1 error: subprocess-exited-with-error × python setup.py develop did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. full command: /mnt/home/lgarrison/miniconda3/envs/nufft-ls-py39/bin/python -c ' exec(compile('"'"''"'"''"'"' # This is -- a caller that pip uses to run setup.py # # - It imports setuptools before invoking setup.py, to enable projects that directly # import from `distutils.core` to work with newer packaging standards. # - It provides a clear error message when setuptools is not installed. # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so # setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning: # manifest_maker: standard file '"'"'-c'"'"' not found". # - It generates a shim setup.py, for handling setup.cfg-only projects. import os, sys, tokenize try: import setuptools except ImportError as error: print( "ERROR: Can not execute `setup.py` since setuptools is not available in " "the build environment.", file=sys.stderr, ) sys.exit(1) __file__ = %r sys.argv[0] = __file__ if os.path.exists(__file__): filename = __file__ with tokenize.open(__file__) as f: setup_py_code = f.read() else: filename = "" setup_py_code = "from setuptools import setup; setup()" exec(compile(setup_py_code, filename, "exec")) '"'"''"'"''"'"' % ('"'"'/mnt/home/lgarrison/nufft-ls/setup.py'"'"',), "", "exec"))' develop --no-deps cwd: /mnt/home/lgarrison/nufft-ls/ error: subprocess-exited-with-error × python setup.py develop did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. (nufft-ls-py39) lgarrison@ccalin008:~/nufft-ls$ ```

@dfm any ideas? I've looked around for similar issue reports but can't find any. I'll try again from scratch with a dummy pybind11 project and see if the error persists...

Interestingly, pip install pybind11, which solved #7, does not solve the problem for me, nor does --no-build-isolation.

[dfm]

I'll take a look!

[dfm]

@lgarrison: any reason why you switched to using intree_extensions? The following diff (reverting a change you made) fixes it for me:

- ext_modules = intree_extensions(["src/nufft_ls/cpu.cpp"])
- ext_modules[0].include_dirs = ['include/']
- ext_modules[0].language = 'c++'
+ ext_modules = [
+     Pybind11Extension(
+         "nufft_ls.cpu", ["src/nufft_ls/cpu.cpp"], include_dirs=["include"]
+     )
+ ]

[lgarrison]

No particular reason! I was mostly just trying it because the docs suggested it was an option when you want to lay out your source files this way. (And I know I tried reverting that change while debugging this... will confirm the fix asap!)

And I guess I thought that those two formulations were supposed to be equivalent! Any idea why one would work and the other not? (To be clear, both work inside a venv for me)

[dfm]

Yeah the docs make it sound like they should be but perhaps the issue is with manually setting the include_dirs or something?

[lgarrison]

Ah, yeah, that's exactly what it was. I should have been appending to the include_dirs. Thanks!