search

DedSecInside / gotor

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.
GNU General Public License v3.0
158 stars 44 forks source link

Add concurrency model to link queries #17

Closed KingAkeem closed 3 years ago

KingAkeem commented 3 years ago

Currently all links are parsed before any querying of status occurs. Another issue is that each HTTP request occurs synchronously.

I'm going to benchmark the current synchronous model against an asynchronous model that allows the links to retrieve their status as soon as they're parsed instead of waiting for the entire parsing to happen first.