search

allenai / objaverse-xl

🪐 Objaverse-XL is a Universe of 10M+ 3D Objects. Contains API Scripts for Downloading and Processing!
https://objaverse.allenai.org/
Apache License 2.0
660 stars 37 forks source link

429 Too Many Requests downloading thingiverse data #37

Open eliphatfs opened 3 months ago

eliphatfs commented 3 months ago

With only a single thread set, I'm still getting too many requests after downloading only a few (~100) objects. It seems impossible to download 3M thingiverse data in sensible time.

nallath commented 1 month ago

Sorry for that, but the huge amount of requests that scrapers like this are causing us have forced us to implement rather strict rate limiting. I was already wondering why I was seeing so many scrapers coming from universities as of a few months.