Question: Crawling of onion domains found in pastes automatically feeded to ail by e.g. pystemon

ail-project / ail-framework

AIL framework - Analysis Information Leak framework

GNU Affero General Public License v3.0

561 stars 79 forks source link

Question: Crawling of onion domains found in pastes automatically feeded to ail by e.g. pystemon #29

Open certrik opened 4 years ago

certrik commented 4 years ago

Regarding the crawling of onion domains found in pastes ... Is it correct that those domains are only crawled once a month and not when they are automatically feeded to ail by e.g pystemon? If this is correct this information would be a good addition to the readme.md .

Terrtia commented 4 years ago

Hi @certrik ! Indeed, domains found by the onions module are only crawled once a month. (new domain are inserted in a priority queue) We should add a crawler page in the documentation.