search

gingeleski / odds-portal-scraper

Sports odds and results scraping for Odds Portal (oddsportal.com).
The Unlicense
104 stars 53 forks source link

Breaking change in dependency joblib #25

Open bbrewington opened 2 years ago

bbrewington commented 2 years ago

in requirements.txt files, joblib==0.13.2 needs to change to joblib==1.1.0, and need to add Cython==0.29.30

Before making the change, got this error:

(venv) ➜  full_scraper git:(master) ✗ python op.py --help
Traceback (most recent call last):
  File "/Users/brentbrewington/project-files-github/odds-portal-scraper/full_scraper/op.py", line 8, in <module>
    from joblib import delayed
  File "/Users/brentbrewington/project-files-github/odds-portal-scraper/full_scraper/venv/lib/python3.9/site-packages/joblib/__init__.py", line 119, in <module>
    from .parallel import Parallel
  File "/Users/brentbrewington/project-files-github/odds-portal-scraper/full_scraper/venv/lib/python3.9/site-packages/joblib/parallel.py", line 28, in <module>
    from ._parallel_backends import (FallbackToBackend, MultiprocessingBackend,
  File "/Users/brentbrewington/project-files-github/odds-portal-scraper/full_scraper/venv/lib/python3.9/site-packages/joblib/_parallel_backends.py", line 22, in <module>
    from .executor import get_memmapping_executor
  File "/Users/brentbrewington/project-files-github/odds-portal-scraper/full_scraper/venv/lib/python3.9/site-packages/joblib/executor.py", line 14, in <module>
    from .externals.loky.reusable_executor import get_reusable_executor
  File "/Users/brentbrewington/project-files-github/odds-portal-scraper/full_scraper/venv/lib/python3.9/site-packages/joblib/externals/loky/__init__.py", line 12, in <module>
    from .backend.reduction import set_loky_pickler
  File "/Users/brentbrewington/project-files-github/odds-portal-scraper/full_scraper/venv/lib/python3.9/site-packages/joblib/externals/loky/backend/reduction.py", line 125, in <module>
    from joblib.externals import cloudpickle  # noqa: F401
  File "/Users/brentbrewington/project-files-github/odds-portal-scraper/full_scraper/venv/lib/python3.9/site-packages/joblib/externals/cloudpickle/__init__.py", line 3, in <module>
    from .cloudpickle import *
  File "/Users/brentbrewington/project-files-github/odds-portal-scraper/full_scraper/venv/lib/python3.9/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 152, in <module>
    _cell_set_template_code = _make_cell_set_template_code()
  File "/Users/brentbrewington/project-files-github/odds-portal-scraper/full_scraper/venv/lib/python3.9/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 133, in _make_cell_set_template_code
    return types.CodeType(
TypeError: an integer is required (got type bytes)

After making the change the error went away:

(venv) ➜  full_scraper git:(master) ✗ python op.py --help            
usage: op.py [-h] [--number-of-cpus [NUMBER_OF_CPUS]]
             [--wait-time-on-page-load [WAIT_TIME_ON_PAGE_LOAD]]

oddsporter v1.0

optional arguments:
  -h, --help            show this help message and exit
  --number-of-cpus [NUMBER_OF_CPUS]
                        Number parallel CPUs for processing (default -1 for max available)
  --wait-time-on-page-load [WAIT_TIME_ON_PAGE_LOAD]
                        How many seconds to wait on page load (default 3)
bbrewington commented 2 years ago

See: https://github.com/gingeleski/odds-portal-scraper/pull/23