more metrics - Githubissues

rom1504 / img2dataset

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

MIT License

3.74k stars 341 forks source link

more metrics #226

Open rom1504 opened 2 years ago

rom1504 commented 2 years ago

download latency (average and 99p per shard)
dns resolving latency (average and 99p per shard)

rom1504 commented 2 years ago

may help investigate why downloading (of eg laion400m) is 2x slower than before (and cpu usage stays low) I'm suspecting the network settings (number of threads, timeout, rate throttling per domain) couldbe improved