search

ArchiveTeam / grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Other
1.31k stars 129 forks source link

ftp:// crawls crash with AttributeError: 'ListingResponse' object has no attribute 'version' #144

Open ivan opened 5 years ago

ivan commented 5 years ago

Reported on IRC:

ERROR Fatal exception.
Traceback (most recent call last):
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/application/app.py", line 157, in run
    yield from pipeline.process()
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/pipeline/pipeline.py", line 194, in process
    yield from self._process_one_worker()
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/pipeline/pipeline.py", line 215, in _process_one_worker
    task.result()
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/pipeline/pipeline.py", line 119, in process
    item = yield from self.process_one(_worker_id=worker_id)
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/pipeline/pipeline.py", line 103, in process_one
    yield from task.process(item)
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/application/tasks/download.py", line 421, in process
    yield from session.app_session.factory['Processor'].process(session)
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/processor/delegate.py", line 29, in process
    return (yield from processor.process(item_session))
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/processor/ftp.py", line 100, in process
    return (yield from session.process())
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/processor/ftp.py", line 151, in process
    wait_time = yield from self._fetch(request, is_file)
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/processor/ftp.py", line 272, in _fetch
    self._item_session
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/processor/rule.py", line 253, in handle_pre_response
    action = self.consult_pre_response_hook(item_session)
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/processor/rule.py", line 407, in consult_pre_response_hook
    item_session
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/application/hook.py", line 87, in call
    return self._callbacks[name](*args, **kwargs)
  File "/root/gs-venv/lib/python3.7/site-packages/wpull/application/plugin.py", line 49, in wrapper
    return func(*args, **kwargs)
  File "/root/gs-venv/lib/python3.7/site-packages/libgrabsite/wpull_hooks.py", line 461, in handle_pre_response
    if response.version == "ICY":
AttributeError: 'ListingResponse' object has no attribute 'version'
CRITICAL Sorry, Wpull unexpectedly crashed.