If DOWNLOAD_FILES is True in the config, it looks to the load_data_csv for which files to download. However, it's looking at os.path.join(DATA_ROOT,message['data_file']) and DATA_ROOT = '/home/ubuntu/bucket' so S3FS is still required to mount the bucket correctly before downloading the files locally.
ErinWeisbart
commented
1 year ago
If DOWNLOAD_FILES is True in the config, it looks to the load_data_csv for which files to download. However, it's looking at
os.path.join(DATA_ROOT,message['data_file'])andDATA_ROOT = '/home/ubuntu/bucket'so S3FS is still required to mount the bucket correctly before downloading the files locally.It seems we change this behavior.