Importing a project's data from XNAT for each run is a large fraction of each iteration of training runs. This is because on each run the data is downloaded from the server to memory and then discarded when the script exits.
We should implement a caching system so that the downloaded data is stored in an ordered way on the local disk.
The persistence of this data could be defined by.
time: a downloaded dataset is cached for a set amount of time and will be deleted after it expires
size: a cached dataset directory is defined and will not exceed a certain size, new data coming in will prompt the deletion of old data from the cache.
Importing a project's data from XNAT for each run is a large fraction of each iteration of training runs. This is because on each run the data is downloaded from the server to memory and then discarded when the script exits.
We should implement a caching system so that the downloaded data is stored in an ordered way on the local disk.
The persistence of this data could be defined by.
@GSTT-CSC/core tagging for visibility/input.