rom1504 / img2dataset

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

MIT License

3.71k stars 338 forks source link

Open rom1504 opened 1 year ago

rom1504 commented 1 year ago

102

main ideas:

write wip json file every N samples instead of only at the end / clean them up / read them
add metrics like byte read / s

rom1504 commented 1 year ago

this is crutial for further improvement