search

samirettali / tor-spider

A spider for Hidden Services
4 stars 0 forks source link

[Improvement] filter the Input Collector from getting non-onion links #6

Closed ghost closed 4 years ago

ghost commented 4 years ago

Hi,

Hope you are all well !

I think we should filter link stored in MongoDB and to keep only '*.onion' links. At a large scale, it inserts too many classic http links.

Thanks in advance for any insights or inputs on that topic.

Cheers, X

samirettali commented 4 years ago

Hi! Sure, it sounds good, I just commited 04dd3340a4cc22f362e7b1b69212ddb4a54cc610, thanks!