internetarchive / Zeno

State-of-the-art web crawler 🔱
GNU Affero General Public License v3.0
83 stars 11 forks source link

Reevaluate host-based concurrent crawling limits #151

Open NGTmeaty opened 1 month ago

NGTmeaty commented 1 month ago

This limit was in place on our old queue. There is a partial version currently implemented but needs to be evaluated more thoroughly to ensure it is still working correctly.