Closed erow closed 2 years ago
Hi!
We have a dedicated paragraph in the documentation covering this exact use case: https://docs.ffcv.io/parameter_tuning.html#scenario-large-scale-datasets.
TLDR: you shouldn't use RANDOM. Perfect uniform sampling of dataset with small samples will require to almost keep the entire dataset in RAM.
It may be a bug about error information. I find that the disk is out of storage, so the file is incomplete. However, I did not see any alert. It should be better to check the validation of the file before loading it.
The following code does not work:
The error message:
My environment is python3.9 + torch1.7.1 cu10.1 with 64G memory. Is it a wrong setting or a bug?