a solution for .onion forums?

lairdshaw / fups

FUPS: Forum user-post scraper

GNU Affero General Public License v3.0

21 stars 9 forks source link

a solution for .onion forums? #2

Open davisbra opened 6 years ago

davisbra commented 6 years ago

thanks for your project, i tested it and it was good... Is it possible that FUPS extract forum information from .onion domains (tor network) ,too? It seems that it needs to define Tor and a proxy like privoxy/polipo! can you help to extend your project to crawl .onion forums? It makes the project complete

lairdshaw commented 6 years ago

Glad it worked for you. Re TOR, it's not something I know much about, and a higher priority than adding support for TOR is a major refactor of the code so that it has a proper API for scraping forums - an API which can be used in independent apps and which the current user-interface would plug in to. If you or anybody else wants to add support for TOR though, then please do that - I am happy to accept a pull request.