search

sergiotapia / magnetissimo

Web application that indexes all popular torrent sites, and saves it to the local database.
MIT License
2.98k stars 187 forks source link

On demand search is very slow, how do we speed it up? #144

Closed sergiotapia closed 1 year ago

sergiotapia commented 1 year ago

This affects the JSON /search/:search_term endpoint.

Right now we crawl the first page of results from each crawler in parallel.

But if one crawler has 50 torrents we crawl those sequentially. I believe what needs to happen here is we need to grab those 50 urls in search page 1, chunk them and async stream/process them 10 at a time.

I'll figure this out tomorrow.

End result should be a much faster on demand search. We need this otherwise the product sucks.

sergiotapia commented 1 year ago

This is now fixed in:

bcc2f97a1ba299ca80df50b47cb0f252a6cd58ea

f6a281d4c27f622e8960bd705ffe1345a88221ae