search

CIRCL / AIL-framework

AIL framework - Analysis Information Leak framework. Project moved to https://github.com/ail-project
https://github.com/ail-project/ail-framework
GNU Affero General Public License v3.0
1.29k stars 283 forks source link

Pastebin being crawled by tor proxy #552

Closed Skullack closed 5 months ago

Skullack commented 3 years ago

Hi How do I prevent the system from crawling pastebin using the tor proxy.

I have tried the following Turning tor service off and also choosing dont use splash crawler.

If tor serivce is down then nothing is crawled. If tor service run. Pastebin is crawled using tor proxy only causing issues with crawling pastebin seeing that I have to whitelist an IP address which obviously with tor is not whitelisted.

Looking forward to your reply I submitted same on gitter.com but no response from anyone there.

Terrtia commented 5 months ago

AIL is not crawling pastebin. https://github.com/cvandeplas/pystemon