internetarchive / Zeno

State-of-the-art web crawler 🔱
GNU Affero General Public License v3.0
83 stars 11 forks source link

Optimize `get list` loading performance #104

Closed yzqzss closed 4 months ago

yzqzss commented 4 months ago

The time to load 10,000,000 URLs was reduced from 70s to 15s on my machine.

Printing progress for every line is expensive.