search

TorSpider / TorSpider-Backend

The database backend with which the spiders share their discoveries.
BSD 3-Clause "New" or "Revised" License
1 stars 0 forks source link

Transfer the majority of information processing from the spider to the backend. #40

Open haxys opened 6 years ago

haxys commented 6 years ago

Offloading the heavy lifting to the backend allows us to use only two spider-side API queries for processing each url rather than the hundreds it takes currently. All we'll need is one query to get the url to scan, and one query to add the discovered information to the database.

haxys commented 6 years ago

See: https://github.com/TorSpider/TorSpider/issues/35

haxys commented 6 years ago

This should be pretty much complete at this point, with the offload_to_backend pull request. Just test and commit.