Files from `file_cache` are persisting

flatironinstitute / dendro-old

Analyze neuroscience data in the cloud

https://flatironinstitute.github.io/dendro-docs/

Apache License 2.0

19 stars 2 forks source link

Files from `file_cache` are persisting #139

Open luiztauffer opened 4 months ago

luiztauffer commented 4 months ago

The files stored at file_cache are persisting on apptainer mode:

magland commented 4 months ago

I added the file cache in order to enable jobs to use the outputs of previous jobs without needing to re-download. It also enables pipelines where the intermediate files are not uploaded to the cloud.

I suppose we should make some configuration option to either not use this cache, or to automatically clear old files.

luiztauffer commented 4 months ago

yeah that makes sense, probably at the job config level, options like these?

[ ] use cache from previous jobs
[ ] clean cache from this job