Error installing in CI environment (Linux, Python 3.7), `fatal error: Python.h: No such file or directory`

[alxmrs]

Environment

• a GitLab CI with a python:3.8/Linux docker image
• I use tox as a testrunner. This trace below occurs when I install requirements in the py37 environment (but succeeds for py38)

Project requirements

cdsapi
ecmwf-api-client
apache-beam[gcp]
numpy>=1.19.1
pandas
xarray
requests>=2.24.0
firebase-admin>=5.0.0
google-cloud-datastore>=1.15.0,<2  # For compatability with apache-beam[gcp]
google-cloud-firestore
urllib3==1.25.11

Profiler version

Collecting google-cloud-profiler<4,>=3.0.4
  Downloading google-cloud-profiler-3.0.4.tar.gz (32 kB)

Stack trace

Building wheels for collected packages: google-cloud-profiler
  Building wheel for google-cloud-profiler (setup.py): started
  Building wheel for google-cloud-profiler (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /builds/XXXXXX/.tox/py37/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-n8xc8_iv/google-cloud-profiler_67e67791160d41a2bb5cb70e302bb400/setup.py'"'"'; __file__='"'"'/tmp/pip-install-n8xc8_iv/google-cloud-profiler_67e67791160d41a2bb5cb70e302bb400/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-yhglrqsm
       cwd: /tmp/pip-install-n8xc8_iv/google-cloud-profiler_67e67791160d41a2bb5cb70e302bb400/
  Complete output (26 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/client.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/__version__.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/pythonprofiler.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/__init__.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/builder.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/cpu_profiler.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/backoff.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/profile_pb2.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  running build_ext
  building 'googlecloudprofiler._profiler' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/googlecloudprofiler
  creating build/temp.linux-x86_64-3.7/googlecloudprofiler/src
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Igooglecloudprofiler/src -I/builds/XXXXXXX/.tox/py37/include -I/usr/include/python3.7m -c googlecloudprofiler/src/stacktraces.cc -o build/temp.linux-x86_64-3.7/googlecloudprofiler/src/stacktraces.o -std=c++11
  In file included from googlecloudprofiler/src/stacktraces.cc:15:
  googlecloudprofiler/src/stacktraces.h:18:10: fatal error: Python.h: No such file or directory
   #include <Python.h>
            ^~~~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for google-cloud-profiler
  Running setup.py clean for google-cloud-profiler

[wyk9787]

This question is probably what you are looking for. Python.h is not part of the google-cloud-profiler package. I think python3-dev is usually installed with the default Python version. Since you switched to a different Python version, python3.8-dev is probably missing.

[alxmrs]

Thanks! After manually installing python3.7-dev in my CI with apt-get, my error was resolved. Thanks for the pointer.