search

rom1504 / img2dataset

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.
MIT License
3.71k stars 338 forks source link

Respect x-robots-tag directives by default #248

Closed Stealcase closed 1 year ago

Stealcase commented 1 year ago

EDIT: Accidentally made an issue instead of PR

rom1504 commented 1 year ago

https://github.com/rom1504/img2dataset/pull/249 opting out is now enabled by default

If using this default, be aware there are ethical issues with slowing down democratization of skills and art to millions of people.