ArchiveTeam / grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Other
1.32k stars 130 forks source link

ImportError: No module named 'dns.resolver' #95

Closed ivan closed 7 years ago

ivan commented 7 years ago

After upgrading all of grab-site's dependencies with pip3 install --upgrade, the new dnspython (?) breaks grab-site, at least when using Python 3.4.5. The build somehow succeeds on the Travis builder, which is using Python 3.4.2. But I'm not sure the Python version is responsible.

# pip3 install --upgrade git+https://github.com/ludios/grab-site
Collecting git+https://github.com/ludios/grab-site
  Cloning https://github.com/ludios/grab-site to /tmp/pip-0g0jfkaj-build
Requirement already up-to-date: click>=6.3 in /home/grab/gs-venv/lib/python3.4/site-packages (from grab-site==1.1.9)
Requirement already up-to-date: wpull==1.2.3 in /home/grab/gs-venv/lib/python3.4/site-packages (from grab-site==1.1.9)
Requirement already up-to-date: manhole>=1.0.0 in /home/grab/gs-venv/lib/python3.4/site-packages (from grab-site==1.1.9)
Requirement already up-to-date: lmdb>=0.89 in /home/grab/gs-venv/lib/python3.4/site-packages (from grab-site==1.1.9)
Requirement already up-to-date: autobahn>=0.12.1 in /home/grab/gs-venv/lib/python3.4/site-packages (from grab-site==1.1.9)
Requirement already up-to-date: trollius>=2.1 in /home/grab/gs-venv/lib/python3.4/site-packages (from grab-site==1.1.9)
Requirement already up-to-date: html5lib==0.9999999 in /home/grab/gs-venv/lib/python3.4/site-packages (from grab-site==1.1.9)
Requirement already up-to-date: cchardet>=1.0.0 in /home/grab/gs-venv/lib/python3.4/site-packages (from grab-site==1.1.9)
Collecting tornado (from wpull==1.2.3->grab-site==1.1.9)
  Downloading tornado-4.4.2.tar.gz (460kB)
    100% |████████████████████████████████| 462kB 952kB/s 
Requirement already up-to-date: chardet in /home/grab/gs-venv/lib/python3.4/site-packages (from wpull==1.2.3->grab-site==1.1.9)
Collecting sqlalchemy (from wpull==1.2.3->grab-site==1.1.9)
  Downloading SQLAlchemy-1.1.1.tar.gz (5.1MB)
    100% |████████████████████████████████| 5.1MB 111kB/s 
Requirement already up-to-date: namedlist in /home/grab/gs-venv/lib/python3.4/site-packages (from wpull==1.2.3->grab-site==1.1.9)
Collecting dnspython3 (from wpull==1.2.3->grab-site==1.1.9)
  Downloading dnspython3-1.15.0.zip
Requirement already up-to-date: six>=1.10.0 in /home/grab/gs-venv/lib/python3.4/site-packages (from autobahn>=0.12.1->grab-site==1.1.9)
Requirement already up-to-date: txaio>=2.5.1 in /home/grab/gs-venv/lib/python3.4/site-packages (from autobahn>=0.12.1->grab-site==1.1.9)
Requirement already up-to-date: backports-abc>=0.4 in /home/grab/gs-venv/lib/python3.4/site-packages (from tornado->wpull==1.2.3->grab-site==1.1.9)
Collecting dnspython==1.15.0 (from dnspython3->wpull==1.2.3->grab-site==1.1.9)
  Downloading dnspython-1.15.0-py2.py3-none-any.whl (177kB)
    100% |████████████████████████████████| 180kB 2.6MB/s 
Installing collected packages: grab-site, tornado, sqlalchemy, dnspython, dnspython3
  Found existing installation: grab-site 1.1.9
    Uninstalling grab-site-1.1.9:
      Successfully uninstalled grab-site-1.1.9
  Running setup.py install for grab-site
  Found existing installation: tornado 4.4.1
  Running setup.py install for grab-site                                                                                                                                                                                  [48/106]
  Found existing installation: tornado 4.4.1
    Uninstalling tornado-4.4.1:
      Successfully uninstalled tornado-4.4.1
  Running setup.py install for tornado
  Found existing installation: SQLAlchemy 1.0.14
    Uninstalling SQLAlchemy-1.0.14:
      Successfully uninstalled SQLAlchemy-1.0.14
  Running setup.py install for sqlalchemy
  Found existing installation: dnspython 1.14.0
    Uninstalling dnspython-1.14.0:
      Successfully uninstalled dnspython-1.14.0
  Found existing installation: dnspython3 1.12.0
    Uninstalling dnspython3-1.12.0:
      Successfully uninstalled dnspython3-1.12.0
  Running setup.py install for dnspython3
Successfully installed dnspython-1.15.0 dnspython3-1.15.0 grab-site-1.1.9 sqlalchemy-1.1.1 tornado-4.4.2
You are using pip version 7.1.2, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

# grab-site http://fontdeck.com/
Traceback (most recent call last):
  File "/home/grab/gs-venv/bin/grab-site", line 4, in <module>
    main.main()
  File "/home/grab/gs-venv/lib/python3.4/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/grab/gs-venv/lib/python3.4/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/grab/gs-venv/lib/python3.4/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/grab/gs-venv/lib/python3.4/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/grab/gs-venv/lib/python3.4/site-packages/libgrabsite/main.py", line 370, in main
    import wpull.__main__
  File "/home/grab/gs-venv/lib/python3.4/site-packages/wpull/__main__.py", line 9, in <module>
    from wpull.builder import Builder
  File "/home/grab/gs-venv/lib/python3.4/site-packages/wpull/builder.py", line 24, in <module>
    from wpull.connection import Connection, ConnectionPool, SSLConnection
  File "/home/grab/gs-venv/lib/python3.4/site-packages/wpull/connection.py", line 18, in <module>
    from wpull.dns import Resolver
  File "/home/grab/gs-venv/lib/python3.4/site-packages/wpull/dns.py", line 10, in <module>
    import dns.resolver
ImportError: No module named 'dns.resolver'
ivan commented 7 years ago

The problem mysteriously went away after doing

pip3 install dnspython==1.15.0

Maybe pip3 suffered some kind of transient failure and didn't write resolver.py? I have no idea.

If anyone else sees this, please comment/reopen.

kamalakkannan1984 commented 5 years ago

pip install dnspython

Its worked for me