khanlab / SPIMprep

A Snakemake workflow for pre-processing single plane illumination microscopy (SPIM, aka lightsheet microscopy)
MIT License
2 stars 1 forks source link

Add option to write BIDS directly to cloud storage #35

Closed akhanf closed 1 month ago

akhanf commented 4 months ago

This adds a write_to_remote option that if enabled, writes the output directly to cloud storage (s3 or gcs)

It uses a touch file as the output instead of the zarr folder, with the uri passed separately as a param. This is so that an FSStore can be instantiated in the snakemake rule, instead of generating locally then copying.

Had to add a final() function around the outputs that are optionally on remote, and this function adds the remote prefix, along with applying the storage() function.

To do:

akhanf commented 1 month ago

One more thing for me to do: update the QC scripts to use the params.uri if it is remote