search

marklieberman / downloadstar

Download all items in a webpage that match a pattern
GNU General Public License v3.0
90 stars 16 forks source link

Queue size limit should not apply to new items in queue #39

Closed derfasthirnlosenick closed 6 years ago

derfasthirnlosenick commented 6 years ago

While the scraping accepts >1,000 links, the downloads themselves stop after hitting 1,000.

Is there a technical reason why the queue is limited to 1,000 items (download.js, line 325)? If not, would it be possible to increase to, e.g. 500,000?

marklieberman commented 6 years ago

The queue isn't exposed yet, so its all under the hood. I needed to cap it to ensure it doesn't grow without limit and eat a bunch of memory. I didn't think anyone would add 1000+ files in one shot.