search

ArchiveTeam / wpull

Wget-compatible web downloader and crawler.
GNU General Public License v3.0
545 stars 77 forks source link

AttributeError: module 'collections' has no attribute 'Mapping' #475

Open allanlaal opened 1 year ago

allanlaal commented 1 year ago

tried both with pip install and pip install from git:

allan@S4:/afs/acn/src./rasketehnika.ee$ pip3 install wpull
Defaulting to user installation because normal site-packages is not writeable
Collecting wpull
  Downloading wpull-2.0.1.tar.gz (666 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 666.8/666.8 KB 6.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: chardet in /usr/lib/python3/dist-packages (from wpull) (4.0.0)
Collecting dnspython3
  Downloading dnspython3-1.15.0.zip (3.4 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: html5lib in /usr/lib/python3/dist-packages (from wpull) (1.1)
Collecting namedlist
  Downloading namedlist-1.8-py2.py3-none-any.whl (15 kB)
Requirement already satisfied: sqlalchemy in /usr/lib/python3/dist-packages (from wpull) (1.4.31)
Collecting tornado
  Downloading tornado-6.2-cp37-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (423 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 424.0/424.0 KB 17.8 MB/s eta 0:00:00
Requirement already satisfied: yapsy in /usr/lib/python3/dist-packages (from wpull) (1.12.0)
Collecting dnspython==1.15.0
  Downloading dnspython-1.15.0-py2.py3-none-any.whl (177 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 177.4/177.4 KB 12.6 MB/s eta 0:00:00
Requirement already satisfied: greenlet!=0.4.17 in /usr/lib/python3/dist-packages (from sqlalchemy->wpull) (1.1.2)
Building wheels for collected packages: wpull, dnspython3
  Building wheel for wpull (setup.py) ... done
  Created wheel for wpull: filename=wpull-2.0.1-py3-none-any.whl size=698334 sha256=4f26f8251a698b289cd5d358b5ad3adf8ff42f38c44fc9a016787760266e4022
  Stored in directory: /home/allan/.cache/pip/wheels/dc/ca/a1/68ec946890f4b39529488d74e90dd70b2c993bd913082c00fd
  Building wheel for dnspython3 (setup.py) ... done
  Created wheel for dnspython3: filename=dnspython3-1.15.0-py3-none-any.whl size=1773 sha256=2f1983b592c964e01661613e0ef5a3ac64a126f6c17e6f9d43a7bfa55778da11
  Stored in directory: /home/allan/.cache/pip/wheels/74/75/b0/8537bb3aa12042ad362aeec9fb4602de3977ce5266651a7461
Successfully built wpull dnspython3
Installing collected packages: namedlist, dnspython, tornado, dnspython3, wpull
  WARNING: The scripts wpull and wpull3 are installed in '/home/allan/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed dnspython-1.15.0 dnspython3-1.15.0 namedlist-1.8 tornado-6.2 wpull-2.0.1
allan@S4:/afs/acn/src./rasketehnika.ee$ wpull 'https://www.rasketehnika.ee/kasutatud/nimekiri.php?ae=1&ak=0'
Traceback (most recent call last):
  File "/abin/wpull", line 5, in <module>
    from wpull.application.main import main
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/application/main.py", line 4, in <module>
    from wpull.application.builder import Builder
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/application/builder.py", line 8, in <module>
    from wpull.application.tasks.conversion import LinkConversionSetupTask, \
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/application/tasks/conversion.py", line 7, in <module>
    from wpull.pipeline.app import AppSession
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/pipeline/app.py", line 5, in <module>
    from wpull.application.factory import Factory
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/application/factory.py", line 6, in <module>
    class Factory(collections.Mapping, object):
AttributeError: module 'collections' has no attribute 'Mapping'

allan@S4:/afs/acn/src./rasketehnika.ee$ pip3 install git+https://github.com/ArchiveTeam/wpull.git#egg=wpull
Defaulting to user installation because normal site-packages is not writeable
Collecting wpull
  Cloning https://github.com/ArchiveTeam/wpull.git to /tmp/pip-install-3e21e5j3/wpull_2f641b2b6b07404ab32357d4c578fa11
  Running command git clone --filter=blob:none --quiet https://github.com/ArchiveTeam/wpull.git /tmp/pip-install-3e21e5j3/wpull_2f641b2b6b07404ab32357d4c578fa11
  Resolved https://github.com/ArchiveTeam/wpull.git to commit 160d856951769189bb4e1bf2980100ceb961a0a0
  Preparing metadata (setup.py) ... done
Requirement already satisfied: chardet in /usr/lib/python3/dist-packages (from wpull) (4.0.0)
Requirement already satisfied: dnspython3 in /home/allan/.local/lib/python3.10/site-packages (from wpull) (1.15.0)
Requirement already satisfied: html5lib in /usr/lib/python3/dist-packages (from wpull) (1.1)
Requirement already satisfied: namedlist in /home/allan/.local/lib/python3.10/site-packages (from wpull) (1.7)
Requirement already satisfied: sqlalchemy in /home/allan/.local/lib/python3.10/site-packages (from wpull) (1.0.13)
Requirement already satisfied: tornado in /home/allan/.local/lib/python3.10/site-packages (from wpull) (4.5.3)
Requirement already satisfied: yapsy in /home/allan/.local/lib/python3.10/site-packages (from wpull) (1.11.223)
Requirement already satisfied: dnspython==1.15.0 in /home/allan/.local/lib/python3.10/site-packages (from dnspython3->wpull) (1.15.0)
Building wheels for collected packages: wpull
  Building wheel for wpull (setup.py) ... done
  Created wheel for wpull: filename=wpull-2.0.3-py3-none-any.whl size=697762 sha256=b3cf6f23f657273f19ef1d863d9e4996598b182c5cea0e305bb68f80f106b740
  Stored in directory: /tmp/pip-ephem-wheel-cache-d7dfbaau/wheels/4b/99/eb/9ac961bfc18ebb98c933f8dfa57adcb24ef78ee4837ab2aa48
Successfully built wpull
Installing collected packages: wpull
  Attempting uninstall: wpull
    Found existing installation: wpull 2.0.1
    Uninstalling wpull-2.0.1:
      Successfully uninstalled wpull-2.0.1
  WARNING: The scripts wpull and wpull3 are installed in '/home/allan/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed wpull-2.0.3
allan@S4:/afs/acn/src./rasketehnika.ee$ wpull 'https://www.rasketehnika.ee/kasutatud/nimekiri.php?ae=1&ak=0'
Traceback (most recent call last):
  File "/abin/wpull", line 5, in <module>
    from wpull.application.main import main
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/application/main.py", line 4, in <module>
    from wpull.application.builder import Builder
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/application/builder.py", line 8, in <module>
    from wpull.application.tasks.conversion import LinkConversionSetupTask, \
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/application/tasks/conversion.py", line 7, in <module>
    from wpull.pipeline.app import AppSession
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/pipeline/app.py", line 5, in <module>
    from wpull.application.factory import Factory
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/application/factory.py", line 6, in <module>
    class Factory(collections.Mapping, object):
AttributeError: module 'collections' has no attribute 'Mapping'
allan@S4:/afs/acn/src./rasketehnika.ee$ wpull3 'https://www.rasketehnika.ee/kasutatud/nimekiri.php?ae=1&ak=0'
Traceback (most recent call last):
  File "/abin/wpull3", line 5, in <module>
    from wpull.application.main import main
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/application/main.py", line 4, in <module>
    from wpull.application.builder import Builder
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/application/builder.py", line 8, in <module>
    from wpull.application.tasks.conversion import LinkConversionSetupTask, \
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/application/tasks/conversion.py", line 7, in <module>
    from wpull.pipeline.app import AppSession
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/pipeline/app.py", line 5, in <module>
    from wpull.application.factory import Factory
  File "/home/allan/.local/lib/python3.10/site-packages/wpull/application/factory.py", line 6, in <module>
    class Factory(collections.Mapping, object):
AttributeError: module 'collections' has no attribute 'Mapping'
TheTechRobo commented 1 year ago

I think wpull supports python 3.5 max.

allanlaal commented 1 year ago

did not work with 3.5 nor 2.7

TheTechRobo commented 1 year ago

What do you mean by "did not work"? You have to install the dependencies again if you're changing python versions.

JustAnotherArchivist commented 1 year ago

wpull 2.0.3 works fine under Python 3.6 and breaks under 3.7 for unrelated reasons. Python 2 is deader than dead, and wpull hasn't supported it since 2014.

The crash you reported is real though. Python moved the ABCs to collections.abc back in Python 3.3, and the aliases in collections were removed in 3.10.

kelvinhammond commented 1 year ago

On 3.6 I get:

Traceback (most recent call last):
  File "/usr/local/bin/wpull", line 5, in <module>
    from wpull.application.main import main
  File "/usr/local/lib/python3.6/site-packages/wpull/application/main.py", line 4, in <module>
    from wpull.application.builder import Builder
  File "/usr/local/lib/python3.6/site-packages/wpull/application/builder.py", line 12, in <module>
    from wpull.application.tasks.download import ProcessTask, ParserSetupTask, ClientSetupTask, ProcessorSetupTask, \
  File "/usr/local/lib/python3.6/site-packages/wpull/application/tasks/download.py", line 10, in <module>
    from wpull.processor.coprocessor.phantomjs import PhantomJSParams
  File "/usr/local/lib/python3.6/site-packages/wpull/processor/coprocessor/phantomjs.py", line 22, in <module>
    from wpull.processor.rule import ProcessingRule
  File "/usr/local/lib/python3.6/site-packages/wpull/processor/rule.py", line 21, in <module>
    from wpull.protocol.http.robots import RobotsTxtChecker
  File "/usr/local/lib/python3.6/site-packages/wpull/protocol/http/robots.py", line 14, in <module>
    from wpull.protocol.http.web import WebClient
  File "/usr/local/lib/python3.6/site-packages/wpull/protocol/http/web.py", line 13, in <module>
    from wpull.protocol.http.client import Client
  File "/usr/local/lib/python3.6/site-packages/wpull/protocol/http/client.py", line 14, in <module>
    from wpull.protocol.abstract.client import BaseClient, BaseSession, DurationTimeout
  File "/usr/local/lib/python3.6/site-packages/wpull/protocol/abstract/client.py", line 12, in <module>
    from wpull.network.pool import ConnectionPool
  File "/usr/local/lib/python3.6/site-packages/wpull/network/pool.py", line 10, in <module>
    from wpull.network.connection import Connection, SSLConnection
  File "/usr/local/lib/python3.6/site-packages/wpull/network/connection.py", line 13, in <module>
    from tornado.netutil import SSLCertificateError
ImportError: cannot import name 'SSLCertificateError'

Generated with a Dockerfile for python:3.6

FROM python:3.6

WORKDIR /data
RUN pip install --no-cache-dir -U wpull
ENTRYPOINT ["wpull"]
JustAnotherArchivist commented 10 months ago

@kelvinhammond That's a separate issue already filed here: #384