internetarchive / dweb-mirror

Offline Internet Archive project
https://www-dweb-mirror.dev.archive.org/
GNU Affero General Public License v3.0
261 stars 27 forks source link

Crawl and limitTotalTasks #314

Open mitra42 opened 4 years ago

mitra42 commented 4 years ago

Either crawl is ignoring limitTotalTasks or its saying it completed when it didn't .

To test .... try a large crawl (like bali-lontar-transcribed) which is ~12k tasks It will report that it got all of them, and appears to have done so, even though default limit is 3k tasks.