CIRCL / AIL-framework

AIL framework - Analysis Information Leak framework. Project moved to https://github.com/ail-project
https://github.com/ail-project/ail-framework
GNU Affero General Public License v3.0
1.29k stars 283 forks source link

Automatic Crawler | Crawler.py KeyError: 'time' #511

Closed 0vert1m3 closed 5 months ago

0vert1m3 commented 4 years ago

Im using the newest AIL version available on Ubuntu 18.04, if I'm adding a v3 onion domain to the auto crawler with default settings it shows null on the Crawler Dashboard also its stuck on waiting.

Screen output:

------------------START CRAWLER------------------
crawler type:     onion
-------------------------------------------------
url:         http://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.onion
domain:      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.onion
domain_url:  http://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.onion

Traceback (most recent call last):
  File "./Crawler.py", line 413, in <module>
    crawler_config = load_crawler_config(to_crawl['type_service'], url_data['domain'], to_crawl['paste'],  to_crawl['url'], date)
  File "./Crawler.py", line 189, in load_crawler_config
    crawler_config['crawler_options'] = get_crawler_config(redis_crawler, 'auto', service_type, domain, url=url)
  File "./Crawler.py", line 173, in get_crawler_config
    crawler_options['time'] = int(config['time'])
KeyError: 'time'
Terrtia commented 5 months ago

Fixed in v5.0