search

cvdfoundation / open-images-dataset

Open Images is a dataset of ~9 million images that have been annotated with image-level labels and bounding boxes spanning thousands of classes.
https://github.com/openimages/dataset
986 stars 157 forks source link

fatal error:(The read operation timed out) #16

Open huanyuxinchen opened 6 years ago

huanyuxinchen commented 6 years ago

Hi, I tried downloading the train dataset with AWS S3, but every time when I had downloaded almost 140K images, the downloading process would be interrupted, and the error happened: "fatal error:(The read operation timed out)", could you please help me?

DamonLiuTHU commented 6 years ago

may be this problem is caused by the file system, check the limit of the numbers of files per dir.

huanyuxinchen commented 6 years ago

@DamonLiuTHU Thank you for your reply，but I have checked other dir， and I found some dir has more than 400K images，so I think this problem is not caused by this reason.

rabienrose commented 6 years ago

I can only download about 400k images from train folder, than it gave me the error of "fatal error:(The read operation timed out)". it is like there are only about 400k images in the s3 train folder.

huanyuxinchen commented 6 years ago

@rabienrose Me to, I have download 448341 images from train folder, then when I try AWS s3 sync again, the error happened: "fatal error:(The read operation timed out)".

rabienrose commented 6 years ago

@huanyuxinchen I found it recovered this morning, it started downloading the further more files now.

superzrx commented 6 years ago

@rabienrose I have the same problem after downloaded 251k images . The aws just exit by itself, could you please share how it started downloading more files?

huanyuxinchen commented 6 years ago

@rabienrose I still cannot download the further more files.

rabienrose commented 6 years ago

I wrote a script to start aws s3 sync again after it exit with error. So it keep trying until the transferring recover.

superzrx commented 6 years ago

@rabienrose It works! Thank you.

huanyuxinchen commented 6 years ago

@rabienrose I still cannot download, can you share your scripts? Thank you!

rabienrose commented 6 years ago

@huanyuxinchen while : do aws s3 --no-sign-request sync s3://open-images-dataset/train ./train done

huanyuxinchen commented 6 years ago

@rabienrose Thank you, I will try!