jackroos / VL-BERT

Code for ICLR 2020 paper "VL-BERT: Pre-training of Generic Visual-Linguistic Representations".
MIT License
735 stars 110 forks source link

script for downloading GCC images #88

Open sesmae opened 2 years ago

sesmae commented 2 years ago

Hi, Thank you for providing a script for downloading the GCC images. I am using your script for downloading a large set of images similar to GCC. However, the script freezes every 30min-1hr and I have to manually stop the process and run the script again. Because of the freezing it took 2 days to process only 300k urls out of a list of 1.6M so far. Is there any fix to avoid freezing? Thank you for any help in advance.