PaulMcInnis / JobFunnel

Scrape job websites into a single spreadsheet with no duplicates.
MIT License
1.78k stars 210 forks source link

Encountered nasty error message attempting to run `funnel load` in virtualenv... #161

Closed anoduck closed 1 year ago

anoduck commented 1 year ago

Description

Caught this nasty error while attempting to execute jobfunnel inside of a virtual env.

Steps to Reproduce

  1. Clone my repo (don't judge my code, it is scrapy): git clone https://github.com/anoduck/jobfunnel
  2. Switch to dev branch: git checkout dev
  3. Setup project with either poetry or pipenv (both resulted in the same output)
  4. IFF poetry is chosen execute the following code for poetry to load deps from requirements: for item in $(cat requirements.txt); do poetry add "${item}"; done . Else, IFF pipenv, a simple pipenv install should do it.
  5. Initialize the virtualenv with: ("pipenv" or "poetry") shell
  6. Install jobfunnel with python setup.py install
  7. Attempt to execute jobfunnel: /path/to/funnel/bin/funnel load -s my_settings.yml

Expected behavior

Previously, I was able to execute jobfunnel because it was installed as a regular user with python 3.10. Since, python has been updated to 3.11, and python-3.11 does not recognize jobfunnel as being installed. I expected the same configuration, but due to pep 668, this is no longer available without passing an uncomforting "--break-system-packages" flag.

Actual behavior

Jobfunnel now dumps nasty error message when attempted to be run from within a virtual env. The message is as follows.

For poetry:

Traceback (most recent call last):
  File "/home/user/.cache/pypoetry/virtualenvs/jobfunnel-E4eJ2iH_-py3.11/bin/funnel", line 33, in <module>
    sys.exit(load_entry_point('JobFunnel==3.0.2', 'console_scripts', 'funnel')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/pypoetry/virtualenvs/jobfunnel-E4eJ2iH_-py3.11/bin/funnel", line 25, in importlib_load_entry_point
    return next(matches).load()
           ^^^^^^^^^^^^^
StopIteration

For pipenv:

Traceback (most recent call last):
  File "/home/user/.local/share/virtualenvs/jobfunnel-6hvIWohq/bin/funnel", line 33, in <module>
    sys.exit(load_entry_point('JobFunnel==3.0.2', 'console_scripts', 'funnel')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/virtualenvs/jobfunnel-6hvIWohq/bin/funnel", line 25, in importlib_load_entry_point
    return next(matches).load()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/user/.local/share/virtualenvs/jobfunnel-6hvIWohq/lib/python3.11/site-packages/JobFunnel-3.0.2-py3.11.egg/jobfunnel/__main__.py", line 6, in <module>
  File "/home/user/.local/share/virtualenvs/jobfunnel-6hvIWohq/lib/python3.11/site-packages/JobFunnel-3.0.2-py3.11.egg/jobfunnel/backend/__init__.py", line 1, in <module>
  File "/home/user/.local/share/virtualenvs/jobfunnel-6hvIWohq/lib/python3.11/site-packages/JobFunnel-3.0.2-py3.11.egg/jobfunnel/backend/job.py", line 10, in <module>
  File "/home/user/.local/share/virtualenvs/jobfunnel-6hvIWohq/lib/python3.11/site-packages/JobFunnel-3.0.2-py3.11.egg/jobfunnel/resources/__init__.py", line 1, in <module>
  File "/home/user/.local/share/virtualenvs/jobfunnel-6hvIWohq/lib/python3.11/site-packages/JobFunnel-3.0.2-py3.11.egg/jobfunnel/resources/resources.py", line 34, in <module>
NotADirectoryError: [Errno 20] Not a directory:
  '/home/user/.local/share/virtualenvs/jobfunnel-6hvIWohq/lib/python3.11/site-packages/JobFunnel-3.0.2-py3.11.egg/jobfunnel/resources/user_agent_list.txt'

Environment

anoduck commented 1 year ago

Embarrassingly, the error arose from conflicting libraries in a localized python environment, thus not reproducible, and since has been resolved.