search

rom1504 / img2dataset

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.
MIT License
3.76k stars 341 forks source link

Run img2dataset on goolge cloud #375

Closed zwzhu-d closed 11 months ago

zwzhu-d commented 11 months ago

I'm running img2dataset to download Laion400M on Google Cloud. But it is flagged as dos attack. See the info below.

We are notifying you that your project My First Project (id: cellular-unity-410221) has been suspended because it was committing denial of service (DoS) attacks.

If there any solution to downloading with cloud service? Thank you!

rom1504 commented 11 months ago

You can maybe reduce the download rates or use more nodes

But ultimately it's on google cloud to decide what they consider is acceptable. You can ask them maybe

On Fri, Jan 5, 2024, 7:45 PM Zhaowei Zhu @.***> wrote:

I'm running img2dataset to download Laion400M on Google Cloud. But it is flagged as dos attack. See the info below.

We are notifying you that your project My First Project (id: cellular-unity-410221) has been suspended because it was committing denial of service (DoS) attacks.

If there any solution to downloading with cloud service? Thank you!

— Reply to this email directly, view it on GitHub https://github.com/rom1504/img2dataset/issues/375, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAR437XP7R3EBMK5LP5LXYTYNBC5ZAVCNFSM6AAAAABBO2FYAGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA3DOOBXGMYDEMQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>