search

PaulMcInnis / JobFunnel

Scrape job websites into a single spreadsheet with no duplicates.
MIT License
1.78k stars 210 forks source link

Able to run with incorrect version of Python [Ubuntu 16] #125

Closed paul-ojm closed 3 years ago

paul-ojm commented 3 years ago

Issue Template

Description

Getting the following error when running funnel load -s my_settings.yaml SyntaxError: invalid syntax.

Steps to Reproduce

~/python/JobFunnel-3.0.1$ funnel load -s my_settings.yaml Traceback (most recent call last): File "/home/bitnami/.local/bin/funnel", line 5, in from jobfunnel.main import main File "/home/bitnami/.local/lib/python3.5/site-packages/jobfunnel/main.py", line 6, in from .backend.jobfunnel import JobFunnel File "/home/bitnami/.local/lib/python3.5/site-packages/jobfunnel/backend/jobfunnel.py", line 296 f"{cache_file} not found! Have you scraped any jobs today?" ^ SyntaxError: invalid syntax

PaulMcInnis commented 3 years ago

it would seem that you are not using the required version of python.

That said we should check for python version and exit earlier.

paul-ojm commented 3 years ago

bitnami@ip-172-31-20-203:~/python/JobFunnel-3.0.1$ python3 -V Python 3.8.6 bitnami@ip-172-31-20-203:~/python/JobFunnel-3.0.1$ funnel load -s my_settings.yaml Traceback (most recent call last): File "/home/bitnami/.local/bin/funnel", line 5, in from jobfunnel.main import main ModuleNotFoundError: No module named 'jobfunnel' bitnami@ip-172-31-20-203:~/python/JobFunnel-3.0.1$

PaulMcInnis commented 3 years ago

JobFunnel requires Python 3.8 or later.

paul-ojm commented 3 years ago

bitnami@ip-172-31-20-203:~/python/JobFunnel-3.0.1$ funnel load -s my_settings.yaml --recover usage: funnel [-h] [--recover] {load,inline} ... funnel: error: unrecognized arguments: --recover bitnami@ip-172-31-20-203:~/python/JobFunnel-3.0.1$ funnel --recover load -s my_settings.yaml [2020-11-27 18:02:47,277] [INFO] JobFunnel: Recovering jobs from all cache files in cache folder [2020-11-27 18:02:47,278] [DEBUG] JobFunnel: Wrote 0 jobs to demo_job_search_results/demo_search.csv bitnami@ip-172-31-20-203:~/python/JobFunnel-3.0.1$ less demo_job_search_results/demo_search.csv bitnami@ip-172-31-20-203:~/python/JobFunnel-3.0.1$ funnel load -s my_settings.yaml [2020-11-27 18:03:06,015] [DEBUG] JobFunnel: Read 0 jobs from master-CSV: demo_job_search_results/demo_search.csv [2020-11-27 18:03:06,015] [DEBUG] JobFunnel: No master-CSV present, did not update block-list: demo_job_search_results/demo_block_list.json [2020-11-27 18:03:06,016] [INFO] JobFunnel: Scraping local providers with: ['IndeedScraperCANEng', 'MonsterScraperCANEng'] [2020-11-27 18:03:06,290] [DEBUG] IndeedScraperCANEng: Got Base search results page: https://www.indeed.ca/jobs?q=Python&l=toronto%2C+ON&radius=100&limit=50&filter=0 Traceback (most recent call last): File "/home/bitnami/.local/lib/python3.9/site-packages/jobfunnel/backend/scrapers/base.py", line 196, in scrape job_soups = self.get_job_soups_from_search_result_listings() File "/home/bitnami/.local/lib/python3.9/site-packages/jobfunnel/backend/scrapers/indeed.py", line 125, in get_job_soups_from_search_result_listings pages = self._get_num_search_result_pages(search_url) File "/home/bitnami/.local/lib/python3.9/site-packages/jobfunnel/backend/scrapers/indeed.py", line 307, in _get_num_search_result_pages raise ValueError( ValueError: Unable to identify number of pages of results for query: https://www.indeed.ca/jobs?q=Python&l=toronto%2C+ON&radius=100&limit=50&filter=0 Please ensure linked page contains results, you may have provided a city for which there are no results within this province or state.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/bitnami/.local/bin/funnel", line 8, in sys.exit(main()) File "/home/bitnami/.local/lib/python3.9/site-packages/jobfunnel/main.py", line 28, in main job_funnel.run() File "/home/bitnami/.local/lib/python3.9/site-packages/jobfunnel/backend/jobfunnel.py", line 114, in run scraped_jobs_dict = self.scrape() File "/home/bitnami/.local/lib/python3.9/site-packages/jobfunnel/backend/jobfunnel.py", line 236, in scrape incoming_jobs_dict = scraper.scrape() File "/home/bitnami/.local/lib/python3.9/site-packages/jobfunnel/backend/scrapers/base.py", line 198, in scrape raise ValueError( ValueError: Unable to extract jobs from initial search result page: Unable to identify number of pages of results for query: https://www.indeed.ca/jobs?q=Python&l=toronto%2C+ON&radius=100&limit=50&filter=0 Please ensure linked page contains results, you may have provided a city for which there are no results within this province or state.

PaulMcInnis commented 3 years ago

can I see output of pip3 freeze or whatever your pip is that you are running jobfunnel with?

paul-ojm commented 3 years ago

bitnami@ip-172-31-20-203:~/python/indeedjobsearch$ pip3.9 freeze attrs==20.3.0 beautifulsoup4==4.9.3 bs4==0.0.1 Cerberus==1.3.2 certifi==2020.11.8 chardet==3.0.4 click==7.1.2 colorama==0.4.4 configparser==5.0.1 crayons==0.4.0 idna==2.10 indeedjobsearch==6.2.0 iniconfig==1.1.1 JobFunnel @ git+https://github.com/PaulMcInnis/JobFunnel.git@cf80740a852f478b3a656f612136ddcf841af096 joblib==0.17.0 lxml==4.6.2 nltk==3.5 numpy==1.19.4 packaging==20.4 pandas==1.1.4 pluggy==0.13.1 py==1.9.0 pycurl==7.43.0 pygobject==3.20.0 pyparsing==2.4.7 pytest==6.1.2 pytest-mock==3.3.1 python-apt==1.1.0b1+ubuntu0.16.4.9 python-dateutil==2.8.1 pytz==2020.4 PyYAML==5.3.1 regex==2020.11.13 requests==2.25.0 scipy==1.5.4 selenium==3.141.0 six==1.15.0 soupsieve==2.0.1 toml==0.10.2 tqdm==4.54.0 unattended-upgrades==0.1 urllib3==1.26.2 webdriver-manager==3.2.2

PaulMcInnis commented 3 years ago

Are you able to get results with the default settings per instructions in readme?

paul-ojm commented 3 years ago

Seems like there are bigger issues......

bitnami@ip-172-31-20-203:~/python/indeedjobsearch/USA$ pip install git+https://github.com/PaulMcInnis/JobFunnel.git Traceback (most recent call last): File "/home/bitnami/.local/bin/pip", line 5, in from pip._internal.cli.main import main ModuleNotFoundError: No module named 'pip' bitnami@ip-172-31-20-203:~/python/indeedjobsearch/USA$ pip3.9 install git+https://github.com/PaulMcInnis/JobFunnel.git Defaulting to user installation because normal site-packages is not writeable Collecting git+https://github.com/PaulMcInnis/JobFunnel.git Cloning https://github.com/PaulMcInnis/JobFunnel.git to /tmp/pip-req-build-talaxf1t Requirement already satisfied: beautifulsoup4>=4.6.3 in /home/bitnami/.local/lib/python3.9/site-packages (from JobFunnel==3.0.1) (4.9.3) Requirement already satisfied: lxml>=4.2.4 in /home/bitnami/.local/lib/python3.9/site-packages (from JobFunnel==3.0.1) (4.6.2) Requirement already satisfied: requests>=2.19.1 in /home/bitnami/.local/lib/python3.9/site-packages (from JobFunnel==3.0.1) (2.25.0) Requirement already satisfied: python-dateutil>=2.8.0 in /home/bitnami/.local/lib/python3.9/site-packages (from JobFunnel==3.0.1) (2.8.1) Requirement already satisfied: PyYAML>=5.1 in /home/bitnami/.local/lib/python3.9/site-packages (from JobFunnel==3.0.1) (5.3.1) Requirement already satisfied: nltk>=3.4.1 in /home/bitnami/.local/lib/python3.9/site-packages (from JobFunnel==3.0.1) (3.5) Requirement already satisfied: scipy>=1.4.1 in /home/bitnami/.local/lib/python3.9/site-packages (from JobFunnel==3.0.1) (1.5.4) Requirement already satisfied: pytest>=5.3.1 in /home/bitnami/.local/lib/python3.9/site-packages (from JobFunnel==3.0.1) (6.1.2) Requirement already satisfied: pytest-mock>=3.1.1 in /home/bitnami/.local/lib/python3.9/site-packages (from JobFunnel==3.0.1) (3.3.1) Requirement already satisfied: selenium>=3.141.0 in /home/bitnami/.local/lib/python3.9/site-packages (from JobFunnel==3.0.1) (3.141.0) Requirement already satisfied: webdriver-manager>=2.4.0 in /home/bitnami/.local/lib/python3.9/site-packages (from JobFunnel==3.0.1) (3.2.2) Requirement already satisfied: Cerberus>=1.3.2 in /home/bitnami/.local/lib/python3.9/site-packages (from JobFunnel==3.0.1) (1.3.2) Requirement already satisfied: tqdm>=4.47.0 in /home/bitnami/.local/lib/python3.9/site-packages (from JobFunnel==3.0.1) (4.54.0) Requirement already satisfied: soupsieve>1.2 in /home/bitnami/.local/lib/python3.9/site-packages (from beautifulsoup4>=4.6.3->JobFunnel==3.0.1) (2.0.1) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/bitnami/.local/lib/python3.9/site-packages (from requests>=2.19.1->JobFunnel==3.0.1) (1.26.2) Requirement already satisfied: certifi>=2017.4.17 in /home/bitnami/.local/lib/python3.9/site-packages (from requests>=2.19.1->JobFunnel==3.0.1) (2020.11.8) Requirement already satisfied: idna<3,>=2.5 in /home/bitnami/.local/lib/python3.9/site-packages (from requests>=2.19.1->JobFunnel==3.0.1) (2.10) Requirement already satisfied: chardet<4,>=3.0.2 in /home/bitnami/.local/lib/python3.9/site-packages (from requests>=2.19.1->JobFunnel==3.0.1) (3.0.4) Requirement already satisfied: six>=1.5 in /home/bitnami/.local/lib/python3.9/site-packages (from python-dateutil>=2.8.0->JobFunnel==3.0.1) (1.15.0) Collecting scikit-learn>=0.21.2 Using cached scikit-learn-0.23.2.tar.gz (7.2 MB) Installing build dependencies ... error ERROR: Command errored out with exit status 1: command: /usr/bin/python3.9 /usr/local/lib/python3.9/dist-packages/pip-20.3b1-py3.9.egg/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-dwelbm6v/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.28.5' 'numpy==1.13.3; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"' and platform_python_implementation == '"'"'CPython'"'"'' 'numpy==1.14.0; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"' and platform_python_implementation != '"'"'CPython'"'"'' 'numpy==1.14.5; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.6'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'scipy>=0.19.1' cwd: None Complete output (733 lines): Ignoring numpy: markers 'python_version == "3.6" and platform_system != "AIX" and platform_python_implementation == "CPython"' don't match your environment Ignoring numpy: markers 'python_version == "3.6" and platform_system != "AIX" and platform_python_implementation != "CPython"' don't match your environment Ignoring numpy: markers 'python_version == "3.7" and platform_system != "AIX"' don't match your environment Ignoring numpy: markers 'python_version == "3.6" and platform_system == "AIX"' don't match your environment Ignoring numpy: markers 'python_version == "3.7" and platform_system == "AIX"' don't match your environment Ignoring numpy: markers 'python_version >= "3.8" and platform_system == "AIX"' don't match your environment Collecting setuptools Using cached setuptools-50.3.2-py3-none-any.whl (785 kB) Collecting wheel Using cached wheel-0.35.1-py2.py3-none-any.whl (33 kB) Collecting Cython>=0.28.5 Using cached Cython-0.29.21-cp39-cp39-manylinux1_x86_64.whl (1.9 MB) Collecting numpy==1.17.3 Using cached numpy-1.17.3.zip (6.4 MB) Collecting scipy>=0.19.1 Using cached scipy-1.5.4-cp39-cp39-manylinux1_x86_64.whl (25.8 MB) Building wheels for collected packages: numpy Building wheel for numpy (setup.py): started Building wheel for numpy (setup.py): finished with status 'error' ERROR: Command errored out with exit status 1: command: /usr/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/setup.py'"'"'; file='"'"'/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-qs7ge1b5 cwd: /tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/ Complete output (345 lines): Running from numpy source directory. /usr/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'project_urls' warnings.warn(msg) /usr/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'python_requires' warnings.warn(msg) blas_opt_info: blas_mkl_info: customize UnixCCompiler libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu'] NOT AVAILABLE

blis_info:
customize UnixCCompiler
  libraries blis not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

openblas_info:
customize UnixCCompiler
customize UnixCCompiler
  libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
customize UnixCCompiler
  libraries tatlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

atlas_3_10_blas_info:
customize UnixCCompiler
  libraries satlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
customize UnixCCompiler
  libraries ptf77blas,ptcblas,atlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

atlas_blas_info:
customize UnixCCompiler
  libraries f77blas,cblas,atlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

accelerate_info:
  NOT AVAILABLE

/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/system_info.py:690: UserWarning:
    Optimized (vendor) Blas libraries are not found.
    Falls back to netlib Blas library which has worse performance.
    A better performance should be easily gained by switching
    Blas library.
  self.calc_info()
blas_info:
customize UnixCCompiler
  libraries blas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/system_info.py:690: UserWarning:
    Blas (http://www.netlib.org/blas/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [blas]) or by setting
    the BLAS environment variable.
  self.calc_info()
blas_src_info:
  NOT AVAILABLE

/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/system_info.py:690: UserWarning:
    Blas (http://www.netlib.org/blas/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [blas_src]) or by setting
    the BLAS_SRC environment variable.
  self.calc_info()
  NOT AVAILABLE

/bin/sh: 1: svnversion: not found
non-existing path in 'numpy/distutils': 'site.cfg'
lapack_opt_info:
lapack_mkl_info:
customize UnixCCompiler
  libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

openblas_lapack_info:
customize UnixCCompiler
customize UnixCCompiler
  libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

openblas_clapack_info:
customize UnixCCompiler
customize UnixCCompiler
  libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

flame_info:
customize UnixCCompiler
  libraries flame not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

atlas_3_10_threads_info:
Setting PTATLAS=ATLAS
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/local/lib
customize UnixCCompiler
  libraries tatlas,tatlas not found in /usr/local/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib
customize UnixCCompiler
  libraries tatlas,tatlas not found in /usr/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
customize UnixCCompiler
  libraries tatlas,tatlas not found in /usr/lib/x86_64-linux-gnu
<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
  NOT AVAILABLE

atlas_3_10_info:
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/local/lib
customize UnixCCompiler
  libraries satlas,satlas not found in /usr/local/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib
customize UnixCCompiler
  libraries satlas,satlas not found in /usr/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
customize UnixCCompiler
  libraries satlas,satlas not found in /usr/lib/x86_64-linux-gnu
<class 'numpy.distutils.system_info.atlas_3_10_info'>
  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/local/lib
customize UnixCCompiler
  libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib
customize UnixCCompiler
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
customize UnixCCompiler
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib/x86_64-linux-gnu
<class 'numpy.distutils.system_info.atlas_threads_info'>
  NOT AVAILABLE

atlas_info:
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/local/lib
customize UnixCCompiler
  libraries f77blas,cblas,atlas not found in /usr/local/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib
customize UnixCCompiler
  libraries f77blas,cblas,atlas not found in /usr/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
customize UnixCCompiler
  libraries f77blas,cblas,atlas not found in /usr/lib/x86_64-linux-gnu
<class 'numpy.distutils.system_info.atlas_info'>
  NOT AVAILABLE

lapack_info:
customize UnixCCompiler
  libraries lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/system_info.py:1712: UserWarning:
    Lapack (http://www.netlib.org/lapack/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [lapack]) or by setting
    the LAPACK environment variable.
  if getattr(self, '_calc_info_{}'.format(lapack))():
lapack_src_info:
  NOT AVAILABLE

/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/system_info.py:1712: UserWarning:
    Lapack (http://www.netlib.org/lapack/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [lapack_src]) or by setting
    the LAPACK_SRC environment variable.
  if getattr(self, '_calc_info_{}'.format(lapack))():
  NOT AVAILABLE

/usr/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'project_urls'
  warnings.warn(msg)
/usr/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
/usr/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)
running bdist_wheel
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building py_modules sources
creating build
creating build/src.linux-x86_64-3.9
creating build/src.linux-x86_64-3.9/numpy
creating build/src.linux-x86_64-3.9/numpy/distutils
building library "npymath" sources
get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
customize Gnu95FCompiler
Could not locate executable gfortran
Could not locate executable f95
customize IntelFCompiler
Could not locate executable ifort
Could not locate executable ifc
customize LaheyFCompiler
Could not locate executable lf95
customize PGroupFCompiler
Could not locate executable pgfortran
customize AbsoftFCompiler
Could not locate executable f90
Could not locate executable f77
customize NAGFCompiler
customize VastFCompiler
customize CompaqFCompiler
Could not locate executable fort
customize IntelItaniumFCompiler
Could not locate executable efort
Could not locate executable efc
customize IntelEM64TFCompiler
customize GnuFCompiler
Could not locate executable g77
customize G95FCompiler
Could not locate executable g95
customize PathScaleFCompiler
Could not locate executable pathf95
customize NAGFORCompiler
Could not locate executable nagfor
don't know how to compile Fortran code on platform 'posix'
C compiler: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.9 -c'
x86_64-linux-gnu-gcc: _configtest.c
x86_64-linux-gnu-gcc -pthread _configtest.o -o _configtest
success!
removing: _configtest.c _configtest.o _configtest.o.d _configtest
C compiler: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.9 -c'
x86_64-linux-gnu-gcc: _configtest.c
_configtest.c:1:5: warning: conflicting types for built-in function ‘exp’
 int exp (void);
     ^
x86_64-linux-gnu-gcc -pthread _configtest.o -o _configtest
_configtest.o: In function `main':
/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/_configtest.c:6: undefined reference to `exp'
collect2: error: ld returned 1 exit status
failure.
removing: _configtest.c _configtest.o _configtest.o.d
C compiler: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.9 -c'
x86_64-linux-gnu-gcc: _configtest.c
_configtest.c:1:5: warning: conflicting types for built-in function ‘exp’
 int exp (void);
     ^
x86_64-linux-gnu-gcc -pthread _configtest.o -lm -o _configtest
success!
removing: _configtest.c _configtest.o _configtest.o.d _configtest
creating build/src.linux-x86_64-3.9/numpy/core
creating build/src.linux-x86_64-3.9/numpy/core/src
creating build/src.linux-x86_64-3.9/numpy/core/src/npymath
conv_template:> build/src.linux-x86_64-3.9/numpy/core/src/npymath/npy_math_internal.h
  adding 'build/src.linux-x86_64-3.9/numpy/core/src/npymath' to include_dirs.
conv_template:> build/src.linux-x86_64-3.9/numpy/core/src/npymath/ieee754.c
conv_template:> build/src.linux-x86_64-3.9/numpy/core/src/npymath/npy_math_complex.c
None - nothing done with h_files = ['build/src.linux-x86_64-3.9/numpy/core/src/npymath/npy_math_internal.h']
building library "npysort" sources
creating build/src.linux-x86_64-3.9/numpy/core/src/common
conv_template:> build/src.linux-x86_64-3.9/numpy/core/src/common/npy_sort.h
  adding 'build/src.linux-x86_64-3.9/numpy/core/src/common' to include_dirs.
creating build/src.linux-x86_64-3.9/numpy/core/src/npysort
conv_template:> build/src.linux-x86_64-3.9/numpy/core/src/npysort/quicksort.c
conv_template:> build/src.linux-x86_64-3.9/numpy/core/src/npysort/mergesort.c
conv_template:> build/src.linux-x86_64-3.9/numpy/core/src/npysort/timsort.c
conv_template:> build/src.linux-x86_64-3.9/numpy/core/src/npysort/heapsort.c
conv_template:> build/src.linux-x86_64-3.9/numpy/core/src/npysort/radixsort.c
conv_template:> build/src.linux-x86_64-3.9/numpy/core/src/common/npy_partition.h
conv_template:> build/src.linux-x86_64-3.9/numpy/core/src/npysort/selection.c
conv_template:> build/src.linux-x86_64-3.9/numpy/core/src/common/npy_binsearch.h
conv_template:> build/src.linux-x86_64-3.9/numpy/core/src/npysort/binsearch.c
None - nothing done with h_files = ['build/src.linux-x86_64-3.9/numpy/core/src/common/npy_sort.h', 'build/src.linux-x86_64-3.9/numpy/core/src/common/npy_partition.h', 'build/src.linux-x86_64-3.9/numpy/core/src/common/npy_binsearch.h']
building extension "numpy.core._dummy" sources
Generating build/src.linux-x86_64-3.9/numpy/core/include/numpy/config.h
C compiler: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.9 -c'
x86_64-linux-gnu-gcc: _configtest.c
_configtest.c:1:20: fatal error: Python.h: No such file or directory
compilation terminated.
failure.
removing: _configtest.c _configtest.o
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/setup.py", line 443, in <module>
    setup_package()
  File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/setup.py", line 435, in setup_package
    setup(**metadata)
  File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/core.py", line 171, in setup
    return old_setup(**new_attr)
  File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 179, in run
    self.run_command('build')
  File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/command/build.py", line 47, in run
    old_build.run(self)
  File "/usr/lib/python3.9/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/command/build_src.py", line 142, in run
    self.build_sources()
  File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/command/build_src.py", line 159, in build_sources
    self.build_extension_sources(ext)
  File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/command/build_src.py", line 316, in build_extension_sources
    sources = self.generate_sources(sources, ext)
  File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/command/build_src.py", line 369, in generate_sources
    source = func(extension, build_dir)
  File "numpy/core/setup.py", line 430, in generate_config_h
    moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir)
  File "numpy/core/setup.py", line 49, in check_types
    out = check_types(*a, **kw)
  File "numpy/core/setup.py", line 286, in check_types
    raise SystemError(
SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.
----------------------------------------
ERROR: Failed building wheel for numpy
Running setup.py clean for numpy
ERROR: Command errored out with exit status 1:
 command: /usr/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/setup.py'"'"'; __file__='"'"'/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
     cwd: /tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77
Complete output (10 lines):
Running from numpy source directory.

`setup.py clean` is not supported, use one of the following instead:

  - `git clean -xdf` (cleans all files)
  - `git clean -Xdf` (cleans all versioned files, doesn't touch
                      files that aren't checked into the git repo)

Add `--force` to your command to use it anyway if you must (unsupported).

----------------------------------------
ERROR: Failed cleaning build dir for numpy

Failed to build numpy Installing collected packages: numpy, wheel, setuptools, scipy, Cython Running setup.py install for numpy: started Running setup.py install for numpy: finished with status 'error' ERROR: Command errored out with exit status 1: command: /usr/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/setup.py'"'"'; file='"'"'/tmp/pip-install-zon116aq/numpydba34e690e744051845d8400f189dd77/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-m6n9enr/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-dwelbm6v/overlay --compile --install-headers /tmp/pip-build-env-dwelbm6v/overlay/include/python3.9/numpy cwd: /tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/ Complete output (336 lines): Running from numpy source directory.

  Note: if you need reliable uninstall behavior, then install
  with pip instead of using `setup.py install`:

    - `pip install .`       (from a git repo or downloaded source
                             release)
    - `pip install numpy`   (last NumPy release on PyPi)

  /usr/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'project_urls'
    warnings.warn(msg)
  /usr/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'python_requires'
    warnings.warn(msg)
  blas_opt_info:
  blas_mkl_info:
  customize UnixCCompiler
    libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
    NOT AVAILABLE

  blis_info:
  customize UnixCCompiler
    libraries blis not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
    NOT AVAILABLE

  openblas_info:
  customize UnixCCompiler
  customize UnixCCompiler
    libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
    NOT AVAILABLE

  atlas_3_10_blas_threads_info:
  Setting PTATLAS=ATLAS
  customize UnixCCompiler
    libraries tatlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
    NOT AVAILABLE

  atlas_3_10_blas_info:
  customize UnixCCompiler
    libraries satlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
    NOT AVAILABLE

  atlas_blas_threads_info:
  Setting PTATLAS=ATLAS
  customize UnixCCompiler
    libraries ptf77blas,ptcblas,atlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
    NOT AVAILABLE

  atlas_blas_info:
  customize UnixCCompiler
    libraries f77blas,cblas,atlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
    NOT AVAILABLE

  accelerate_info:
    NOT AVAILABLE

  /tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/system_info.py:690: UserWarning:
      Optimized (vendor) Blas libraries are not found.
      Falls back to netlib Blas library which has worse performance.
      A better performance should be easily gained by switching
      Blas library.
    self.calc_info()
  blas_info:
  customize UnixCCompiler
    libraries blas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
    NOT AVAILABLE

  /tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/system_info.py:690: UserWarning:
      Blas (http://www.netlib.org/blas/) libraries not found.
      Directories to search for the libraries can be specified in the
      numpy/distutils/site.cfg file (section [blas]) or by setting
      the BLAS environment variable.
    self.calc_info()
  blas_src_info:
    NOT AVAILABLE

  /tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/system_info.py:690: UserWarning:
      Blas (http://www.netlib.org/blas/) sources not found.
      Directories to search for the sources can be specified in the
      numpy/distutils/site.cfg file (section [blas_src]) or by setting
      the BLAS_SRC environment variable.
    self.calc_info()
    NOT AVAILABLE

  /bin/sh: 1: svnversion: not found
  non-existing path in 'numpy/distutils': 'site.cfg'
  lapack_opt_info:
  lapack_mkl_info:
  customize UnixCCompiler
    libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
    NOT AVAILABLE

  openblas_lapack_info:
  customize UnixCCompiler
  customize UnixCCompiler
    libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
    NOT AVAILABLE

  openblas_clapack_info:
  customize UnixCCompiler
  customize UnixCCompiler
    libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
    NOT AVAILABLE

  flame_info:
  customize UnixCCompiler
    libraries flame not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
    NOT AVAILABLE

  atlas_3_10_threads_info:
  Setting PTATLAS=ATLAS
  customize UnixCCompiler
    libraries lapack_atlas not found in /usr/local/lib
  customize UnixCCompiler
    libraries tatlas,tatlas not found in /usr/local/lib
  customize UnixCCompiler
    libraries lapack_atlas not found in /usr/lib
  customize UnixCCompiler
    libraries tatlas,tatlas not found in /usr/lib
  customize UnixCCompiler
    libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
  customize UnixCCompiler
    libraries tatlas,tatlas not found in /usr/lib/x86_64-linux-gnu
  <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
    NOT AVAILABLE

  atlas_3_10_info:
  customize UnixCCompiler
    libraries lapack_atlas not found in /usr/local/lib
  customize UnixCCompiler
    libraries satlas,satlas not found in /usr/local/lib
  customize UnixCCompiler
    libraries lapack_atlas not found in /usr/lib
  customize UnixCCompiler
    libraries satlas,satlas not found in /usr/lib
  customize UnixCCompiler
    libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
  customize UnixCCompiler
    libraries satlas,satlas not found in /usr/lib/x86_64-linux-gnu
  <class 'numpy.distutils.system_info.atlas_3_10_info'>
    NOT AVAILABLE

  atlas_threads_info:
  Setting PTATLAS=ATLAS
  customize UnixCCompiler
    libraries lapack_atlas not found in /usr/local/lib
  customize UnixCCompiler
    libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
  customize UnixCCompiler
    libraries lapack_atlas not found in /usr/lib
  customize UnixCCompiler
    libraries ptf77blas,ptcblas,atlas not found in /usr/lib
  customize UnixCCompiler
    libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
  customize UnixCCompiler
    libraries ptf77blas,ptcblas,atlas not found in /usr/lib/x86_64-linux-gnu
  <class 'numpy.distutils.system_info.atlas_threads_info'>
    NOT AVAILABLE

  atlas_info:
  customize UnixCCompiler
    libraries lapack_atlas not found in /usr/local/lib
  customize UnixCCompiler
    libraries f77blas,cblas,atlas not found in /usr/local/lib
  customize UnixCCompiler
    libraries lapack_atlas not found in /usr/lib
  customize UnixCCompiler
    libraries f77blas,cblas,atlas not found in /usr/lib
  customize UnixCCompiler
    libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
  customize UnixCCompiler
    libraries f77blas,cblas,atlas not found in /usr/lib/x86_64-linux-gnu
  <class 'numpy.distutils.system_info.atlas_info'>
    NOT AVAILABLE

  lapack_info:
  customize UnixCCompiler
    libraries lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
    NOT AVAILABLE

  /tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/system_info.py:1712: UserWarning:
      Lapack (http://www.netlib.org/lapack/) libraries not found.
      Directories to search for the libraries can be specified in the
      numpy/distutils/site.cfg file (section [lapack]) or by setting
      the LAPACK environment variable.
    if getattr(self, '_calc_info_{}'.format(lapack))():
  lapack_src_info:
    NOT AVAILABLE

  /tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/system_info.py:1712: UserWarning:
      Lapack (http://www.netlib.org/lapack/) sources not found.
      Directories to search for the sources can be specified in the
      numpy/distutils/site.cfg file (section [lapack_src]) or by setting
      the LAPACK_SRC environment variable.
    if getattr(self, '_calc_info_{}'.format(lapack))():
    NOT AVAILABLE

  /usr/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'project_urls'
    warnings.warn(msg)
  /usr/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'python_requires'
    warnings.warn(msg)
  /usr/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
    warnings.warn(msg)
  running install
  running build
  running config_cc
  unifing config_cc, config, build_clib, build_ext, build commands --compiler options
  running config_fc
  unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
  running build_src
  build_src
  building py_modules sources
  building library "npymath" sources
  get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
  customize Gnu95FCompiler
  Could not locate executable gfortran
  Could not locate executable f95
  customize IntelFCompiler
  Could not locate executable ifort
  Could not locate executable ifc
  customize LaheyFCompiler
  Could not locate executable lf95
  customize PGroupFCompiler
  Could not locate executable pgfortran
  customize AbsoftFCompiler
  Could not locate executable f90
  Could not locate executable f77
  customize NAGFCompiler
  customize VastFCompiler
  customize CompaqFCompiler
  Could not locate executable fort
  customize IntelItaniumFCompiler
  Could not locate executable efort
  Could not locate executable efc
  customize IntelEM64TFCompiler
  customize GnuFCompiler
  Could not locate executable g77
  customize G95FCompiler
  Could not locate executable g95
  customize PathScaleFCompiler
  Could not locate executable pathf95
  customize NAGFORCompiler
  Could not locate executable nagfor
  don't know how to compile Fortran code on platform 'posix'
  C compiler: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

  compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.9 -c'
  x86_64-linux-gnu-gcc: _configtest.c
  x86_64-linux-gnu-gcc -pthread _configtest.o -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest.o.d _configtest
  C compiler: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

  compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.9 -c'
  x86_64-linux-gnu-gcc: _configtest.c
  _configtest.c:1:5: warning: conflicting types for built-in function ‘exp’
   int exp (void);
       ^
  x86_64-linux-gnu-gcc -pthread _configtest.o -o _configtest
  _configtest.o: In function `main':
  /tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/_configtest.c:6: undefined reference to `exp'
  collect2: error: ld returned 1 exit status
  failure.
  removing: _configtest.c _configtest.o _configtest.o.d
  C compiler: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

  compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.9 -c'
  x86_64-linux-gnu-gcc: _configtest.c
  _configtest.c:1:5: warning: conflicting types for built-in function ‘exp’
   int exp (void);
       ^
  x86_64-linux-gnu-gcc -pthread _configtest.o -lm -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest.o.d _configtest
    adding 'build/src.linux-x86_64-3.9/numpy/core/src/npymath' to include_dirs.
  None - nothing done with h_files = ['build/src.linux-x86_64-3.9/numpy/core/src/npymath/npy_math_internal.h']
  building library "npysort" sources
    adding 'build/src.linux-x86_64-3.9/numpy/core/src/common' to include_dirs.
  None - nothing done with h_files = ['build/src.linux-x86_64-3.9/numpy/core/src/common/npy_sort.h', 'build/src.linux-x86_64-3.9/numpy/core/src/common/npy_partition.h', 'build/src.linux-x86_64-3.9/numpy/core/src/common/npy_binsearch.h']
  building extension "numpy.core._dummy" sources
  Generating build/src.linux-x86_64-3.9/numpy/core/include/numpy/config.h
  C compiler: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

  compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.9 -c'
  x86_64-linux-gnu-gcc: _configtest.c
  _configtest.c:1:20: fatal error: Python.h: No such file or directory
  compilation terminated.
  failure.
  removing: _configtest.c _configtest.o
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/setup.py", line 443, in <module>
      setup_package()
    File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/setup.py", line 435, in setup_package
      setup(**metadata)
    File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/core.py", line 171, in setup
      return old_setup(**new_attr)
    File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/command/install.py", line 62, in run
      r = self.setuptools_run()
    File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/command/install.py", line 36, in setuptools_run
      return distutils_install.run(self)
    File "/usr/lib/python3.9/distutils/command/install.py", line 590, in run
      self.run_command('build')
    File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/command/build.py", line 47, in run
      old_build.run(self)
    File "/usr/lib/python3.9/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/command/build_src.py", line 142, in run
      self.build_sources()
    File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/command/build_src.py", line 159, in build_sources
      self.build_extension_sources(ext)
    File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/command/build_src.py", line 316, in build_extension_sources
      sources = self.generate_sources(sources, ext)
    File "/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/numpy/distutils/command/build_src.py", line 369, in generate_sources
      source = func(extension, build_dir)
    File "numpy/core/setup.py", line 430, in generate_config_h
      moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir)
    File "numpy/core/setup.py", line 49, in check_types
      out = check_types(*a, **kw)
    File "numpy/core/setup.py", line 286, in check_types
      raise SystemError(
  SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.
  ----------------------------------------

ERROR: Command errored out with exit status 1: /usr/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zon116aq/numpy_dba34e690e744051845d8400f189dd77/setup.py'"'"'; file='"'"'/tmp/pip-install-zon116aq/numpydba34e690e744051845d8400f189dd77/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-m6n9enr/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-dwelbm6v/overlay --compile --install-headers /tmp/pip-build-env-dwelbm6v/overlay/include/python3.9/numpy Check the logs for full command output.

ERROR: Command errored out with exit status 1: /usr/bin/python3.9 /usr/local/lib/python3.9/dist-packages/pip-20.3b1-py3.9.egg/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-dwelbm6v/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.28.5' 'numpy==1.13.3; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"' and platform_python_implementation == '"'"'CPython'"'"'' 'numpy==1.14.0; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"' and platform_python_implementation != '"'"'CPython'"'"'' 'numpy==1.14.5; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.6'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'scipy>=0.19.1' Check the logs for full command output.

PaulMcInnis commented 3 years ago

Sorry but jobfunnel installs and runs with the default settings per our CI, this seems to be on your end.

Also this is too way too much traceback.

thebigG commented 3 years ago

This is starting to look like an issue with the way python/pip is installed in your system. I think it might help us if you share your OS and how exactly you installed python and pip on your system. It also looks like the issue might be related to user permission, which is something pip, to the best of my knowledge, can be sensitive to if you don't do things in the canonical way.

So if possible, you should post your OS version and exactly how you installed python and pip.

paul-ojm commented 3 years ago

sudo apt-get install python3-pip sudo apt install python3

python is aliased to python3.9 pip3.9 runs fine

pip give the following error. bitnami@ip-172-31-20-203:~/python/indeedjobsearch/USA$ pip Traceback (most recent call last): File "/home/bitnami/.local/bin/pip", line 5, in from pip._internal.cli.main import main File "/usr/lib/python3/dist-packages/pip/init.py", line 14, in from pip.utils import get_installed_distributions, get_prog File "/usr/lib/python3/dist-packages/pip/utils/init.py", line 23, in from pip.locations import ( File "/usr/lib/python3/dist-packages/pip/locations.py", line 9, in from distutils import sysconfig ImportError: cannot import name 'sysconfig' from 'distutils' (/usr/lib/python3.8/distutils/init.py)

pip give the following error

thebigG commented 3 years ago

Could you give us the output for this command: cat /etc/os-release

thebigG commented 3 years ago

Also quick question: Did you alias these things yourself?

paul-ojm commented 3 years ago

Used sudo update-alternatives --config python3 to alias python to python3.9 I did create any alias for pip which is suspect is why you need to run pip3.9 . Disutils was likely not required for 3.9 and was removed causing issues with previous version of pip.

bitnami@ip-172-31-20-203:~/python/indeedjobsearch/Canada$ cat /etc/os-release NAME="Ubuntu" VERSION="16.04.6 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.6 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial

thebigG commented 3 years ago

Let's take it one step at a time.

Run this: funnel --help from your terminal.

And post the output in a comment.

paul-ojm commented 3 years ago

bitnami@ip-172-31-20-203:~/python/indeedjobsearch/Canada$ funnel --help usage: funnel [-h] [--recover] {load,inline} ...

positional arguments: {load,inline} Pass load with a YAML config or inline to pass args by CLI. load Run using an existing configuration YAML. inline Configure search query and data providers via CLI.

optional arguments: -h, --help show this help message and exit --recover Reconstruct a new master CSV file from all available cache files.WARNING: this will replace all the statuses/etc in your master CSV, it is intended for starting fresh / recovering from a bad state.

thebigG commented 3 years ago

Awesome. You have jobfunnel installed. That's good.

Could you run ls -all in that same directory?

paul-ojm commented 3 years ago

bitnami@ip-172-31-20-203:~/python/JobFunnel-3.0.1$ ls -all total 80 drwxr-xr-x 10 bitnami bitnami 4096 Nov 27 17:57 . drwxr-xr-x 4 bitnami bitnami 4096 Nov 27 03:39 .. -rw-r--r-- 1 bitnami bitnami 224 Oct 11 16:42 .codecov.yml drwxr-xr-x 2 bitnami bitnami 4096 Nov 27 17:52 demo drwxr-xr-x 3 bitnami bitnami 4096 Nov 27 18:02 demo_job_search_results drwxr-xr-x 3 bitnami bitnami 4096 Oct 11 16:42 docs drwxr-xr-x 3 bitnami bitnami 4096 Oct 11 16:42 .github -rw-r--r-- 1 bitnami bitnami 2357 Oct 11 16:42 .gitignore drwxr-xr-x 6 bitnami bitnami 4096 Nov 27 03:41 jobfunnel drwxr-xr-x 9 bitnami bitnami 4096 Nov 27 15:51 JobFunnel -rw-r--r-- 1 bitnami bitnami 1069 Oct 11 16:42 LICENSE drwxr-xr-x 2 bitnami bitnami 4096 Oct 11 16:42 logo -rw-r--r-- 1 bitnami bitnami 119 Oct 11 16:42 MANIFEST.in -rw-r--r-- 1 bitnami bitnami 2410 Nov 27 17:57 my_settings.yaml -rw-r--r-- 1 bitnami bitnami 4240 Oct 11 16:42 readme.md -rw-r--r-- 1 bitnami bitnami 237 Oct 11 16:42 requirements.txt -rw-r--r-- 1 bitnami bitnami 1230 Oct 11 16:42 setup.py drwxr-xr-x 5 bitnami bitnami 4096 Oct 11 16:42 tests -rw-r--r-- 1 bitnami bitnami 1640 Oct 11 16:42 .travis.yml

thebigG commented 3 years ago

I meant in the ~/python/indeedjobsearch/Canada directory. Just trying to make sure you don't have a virtual environment running.

In other words, run deactivate.

paul-ojm commented 3 years ago

I do not have virtual env running..... Give me a few mins, I think I may have fixed by re-installing some things.

thebigG commented 3 years ago

Ok. That's a good idea! Make sure you do things in the Canonical way. Any help you need let me know :+1:.

paul-ojm commented 3 years ago

@thebigG Still appears to be the same problem.

thebigG commented 3 years ago

Ok. I will try to reproduce. Give me a few minutes and I will get back to you with feedback.

thebigG commented 3 years ago

It looks like Ubuntu 16 is riddled with Problems when it comes to these newer python versions, but no worries; these steps should work for you :smile: . I started an empty container and these are the steps I took to get JobFunnel running on Ubuntu 16:

1. sudo apt update
2. sudo apt-get install software-properties-common
3. sudo add-apt-repository ppa:deadsnakes/ppa
4. sudo  apt update
5. sudo apt-get install python3.8
6. sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 10
7. sudo apt-get install python3.8-distutils
8. python3 -m easy_install pip
9. sudo apt-get install git
10. sudo apt-get install git
11. sudo apt-get install wget
12. pip install git+https://github.com/PaulMcInnis/JobFunnel.git
13. wget https://git.io/JUWeP -O my_settings.yaml
14. funnel load -s my_settings.yaml

Hopefully these will solve your issues!

paul-ojm commented 3 years ago

@PaulMcInnis @thebigG Working now. Thanks again for all your help. :)

thebigG commented 3 years ago

Awesome! Thanks for reporting the issue. Now we know of these issues in Ubuntu 16. Will be adding a [Ubuntu 16] tag to the title for users that might have the same issue.

PaulMcInnis commented 3 years ago

Seems like this has been resolved. Thanks for reaching out @paul-ojm