In many supercomputers, process of running ML codes is to first run the download part on the login nodes (which have access to the internet), and stop the code right before the actual training starts.
Then, when you run on the compute nodes (the ones with the actual gpus and no internet access), you let the code run to the end. In other frameworks, data downloaders detect the files' presence and skip it before ever trying to connect to the internet.
Flash tries first to check file size in this line, which will freeze in a machine without internet.
To Reproduce
call "download_data" on a machine with no internet access
Code sample
from flash.core.data.utils import download_data
download_data("https://pl-flash-data.s3.amazonaws.com/hymenoptera_data.zip", "data/")
Expected behavior
If the file is already there, skip download
Environment
OS (e.g., Linux): Centos 8.6
Python version: 3.10
PyTorch/Lightning/Flash Version (e.g., 1.10/1.5/0.7): Pytorch 1.12.1, Lightning 0.8.4, Flash 0.8.1.post0
GPU models and configuration: 16x A100 40GB
Any other relevant information:
Additional context
Fast.ai's fastdownload, for example, does not suffer from this - if the file is there, it doesn't try to download it, even if it's of the wrong size: fastdownload link
🐛 Bug
In many supercomputers, process of running ML codes is to first run the download part on the login nodes (which have access to the internet), and stop the code right before the actual training starts.
Then, when you run on the compute nodes (the ones with the actual gpus and no internet access), you let the code run to the end. In other frameworks, data downloaders detect the files' presence and skip it before ever trying to connect to the internet.
Flash tries first to check file size in this line, which will freeze in a machine without internet.
To Reproduce
call "download_data" on a machine with no internet access
Code sample
Expected behavior
If the file is already there, skip download
Environment
Additional context
Fast.ai's fastdownload, for example, does not suffer from this - if the file is there, it doesn't try to download it, even if it's of the wrong size: fastdownload link