CIRCL / AIL-framework

AIL framework - Analysis Information Leak framework. Project moved to https://github.com/ail-project
https://github.com/ail-project/ail-framework
GNU Affero General Public License v3.0
1.29k stars 283 forks source link

TorCrawler: NameError: name 'request' is not defined #526

Closed GaganBhat closed 3 years ago

GaganBhat commented 3 years ago

Crawler, splash and Tor server is running.

Steps to reproduce: Ask to crawl any site (http://google.com/ in this case)

TorSplashCrawler.py says NameError: name 'request' is not defined.

Logs from Crawler_AIL screen:-

File "/home/gaganbhat6/secondary-ail/AIL-framework/AILENV/lib/python3.6/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/gaganbhat6/secondary-ail/AIL-framework/AILENV/lib/python3.6/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "/home/gaganbhat6/secondary-ail/AIL-framework/bin/torcrawler/TorSplashCrawler.py", line 177, in parse
    error_retry = request.meta.get('error_retry', 0)
NameError: name 'request' is not defined
2020-10-04 14:21:55 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1984,
 'downloader/request_count': 1,
 'downloader/request_method_count/POST': 1,
 'downloader/response_bytes': 171,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'elapsed_time_seconds': 0.131612,
 'finish_reason': 'closespider_pagecount',
 'finish_time': datetime.datetime(2020, 10, 4, 14, 21, 55, 332985),
 'log_count/DEBUG': 1,
 'log_count/ERROR': 1,
 'log_count/INFO': 10,
 'log_count/WARNING': 1,
 'memusage/max': 70778880,
 'memusage/startup': 70778880,
 'response_received_count': 1,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'spider_exceptions/NameError': 1,
 'splash/execute/request_count': 1,
 'splash/execute/response_count/200': 1,
 'start_time': datetime.datetime(2020, 10, 4, 14, 21, 55, 201373)}
2020-10-04 14:21:55 [scrapy.core.engine] INFO: Spider closed (closespider_pagecount)