Some imports from standard lib collections are breaking on python 3.10

lbmendes commented 3 years ago

Hi everyone,

Based on an issue from another repo (https://github.com/okfn-brasil/querido-diario/issues/502), I noticed that scrapinghub is using some imports from standard lib collections that are deprecated and not working on Python 3.10.

In Python 3.8 I have these results on ipython console:

In [1]: from collections import Iterator
<ipython-input-1-4fb967d2a9f8>:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  from collections import Iterator

In [2]: from collections import Iterable
<ipython-input-2-c0513a1e6784>:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  from collections import Iterable

In [3]: from collections import MutableMapping
<ipython-input-3-069a7babadbf>:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  from collections import MutableMapping

According to this, it is necessary to change the imports of Iterable, Iterator and MutableMapping to get these items from "collections.abc" instead of just "collections"

Here are the list of imports that I found:

tests/client/test_job.py - from collections import Iterator
tests/client/test_frontiers.py - from collections import Iterable
tests/client/test_projects.py - from collections import defaultdict, Iterator
scrapinghub/hubstorage/resourcetype.py - from collections import MutableMapping

lbmendes commented 3 years ago

It looks that this pull request (https://github.com/scrapinghub/python-scrapinghub/pull/159) covers this issue.

apalala commented 3 years ago

@lbmendes et all. Would you kindly help us test the upgrades to scrapinghub in #159 and shub by pip installing from the Github branches?

lbmendes commented 3 years ago

Hi @apalala I made some import tests using docker. Here are the results:

Tests with scrapinghub

Test 1 - Python 3.9, scrapinghub installed from pip

docker run -ti python:3.9-slim bash -c "pip install scrapinghub; python -c 'import scrapinghub'"

Result: No error

Test 2 - Python 3.10, scrapinghub installed from pip

docker run -ti python:3.10-slim bash -c "pip install scrapinghub; python -c 'import scrapinghub'"

Result: Error described in this issue

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/scrapinghub/__init__.py", line 13, in <module>
    from .hubstorage import HubstorageClient
  File "/usr/local/lib/python3.10/site-packages/scrapinghub/hubstorage/__init__.py", line 6, in <module>
    from .client import HubstorageClient
  File "/usr/local/lib/python3.10/site-packages/scrapinghub/hubstorage/client.py", line 8, in <module>
    from .project import Project
  File "/usr/local/lib/python3.10/site-packages/scrapinghub/hubstorage/project.py", line 3, in <module>
    from .job import Job
  File "/usr/local/lib/python3.10/site-packages/scrapinghub/hubstorage/job.py", line 2, in <module>
    from .resourcetype import (ItemsResourceType, DownloadableResource,
  File "/usr/local/lib/python3.10/site-packages/scrapinghub/hubstorage/resourcetype.py", line 5, in <module>
    from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections' (/usr/local/lib/python3.10/collections/__init__.py)

Test 3 - Python 3.9, scrapinghub installed from github (branch update_py310_minus_py27)

docker run -ti python:3.9-slim bash -c "apt update && apt install git -y; pip install git+https://github.com/scrapinghub/python-scrapinghub.git@update_py310_minus_py27#egg=scrapinghub; python -c 'import scrapinghub'"

Result: No error

Test 4 - Python 3.10, scrapinghub installed from github (branch update_py310_minus_py27)

docker run -ti python:3.10-slim bash -c "apt update && apt install git -y; pip install git+https://github.com/scrapinghub/python-scrapinghub.git@update_py310_minus_py27#egg=scrapinghub; python -c 'import scrapinghub'"

Result: No error :tada: :tada: :tada:

Tests with shub

Test 1 - Python 3.9, shub installed from pip

docker run -ti python:3.9-slim bash -c "pip install shub; python -c 'import shub'"

Result: No error

Test 2 - Python 3.10, shub installed from pip

docker run -ti python:3.10-slim bash -c "pip install shub; python -c 'import shub'"

Result: No error

Test 3 - Python 3.9, shub installed from github (branch update_dependencies)

docker run -ti python:3.9-slim bash -c "apt update && apt install git -y; pip install git+https://github.com/scrapinghub/shub.git@update_dependencies#egg=shub; python -c 'import shub'"

Result: No error

Test 4 - Python 3.10, shub installed from github (branch update_dependencies)

docker run -ti python:3.10-slim bash -c "apt update && apt install git -y; pip install git+https://github.com/scrapinghub/shub.git@update_dependencies#egg=shub; python -c 'import shub'"

Result: No error

apalala commented 3 years ago

Thanks for those tests, @lbmendes. They are like the ones the libraries are running using tox.

Python 3.10 is not the only reason for the upgrade. These libraries were pinning other libraries to very old versions, and that has been fixed too.

Gallaecio commented 2 years ago

Fixed by https://github.com/scrapinghub/python-scrapinghub/pull/166 and released in 2.4.0.

scrapinghub / python-scrapinghub