Simplification proposal

[tsalo]

@akimbler and I noticed something about Singularity images. They seem to automatically mount whatever the current working directory is when they're called, even if that directory isn't in /scratch or /home/[username]. This means that, even if we're in /data, we can call an image from there and interact with the files that are present. If this isn't some kind of loophole, and is an expected behavior, then we can move most of the copying/merging code from cis-processing to cis-bidsify.

Here is a first draft of an outline of the current steps of cis-processing:

0. Submit cis-processing job.
1. Load config file.
2. Make temporary directories in /scratch.
3. Check that heuristic, bidsifier, datalad, and MRIQC files exist.
4. Copy necessary bidsifier, datalad, and MRIQC Singularity images to /scratch.
   • Technically, this step isn't really necessary, as images can be stored wherever. However, using images in /scratch might speed things up.
   • New intermediate step: CD into project directory.
5. Copy raw data tarfile to /scratch. --> cis-bidsify (as long as tarfile is in project directory)
6. Run BIDSifier Singularity image on tarfile.
   • If we output to /data, we can use datalad option in heudiconv.
   • If we output to /scratch we can call datalad within BIDSifier image, after dataset merge.
7. Check if BIDSifier's BIDS-Validator output says the conversion was successful. --> cis-bidsify
8. Merge mini BIDS dataset in /scratch into main BIDS dataset in /data. --> cis-bidsify
   • If we output to /data, we can drop this.
   • If we output to /scratch, we can move dataset merging code to cis-bidsutils.
9. Copy CIS templateflow to /scratch.
10. Run MRIQC Singularity image on new mini-BIDS dataset.
11. Merge MRIQC derivatives in /scratch into main derivatives folder in /data. --> cis-bidsify?
    • If we use cis-bidsify, we can move merging function to cis-bidsutils.
12. Clean up working directory in /scratch.

[tsalo]

This leads to the following changes:

• cis-processing
  • CD to project's root folder at beginning (e.g., /home/data/nbc/Sutherland_ACE).
  • Remove merging functions.
• cis-bidsify
  • Add working directory option. Working directory must be in project root folder or scratch.
    • This allows us to use the project's dset folder as the output folder without making temporary subfolders in the dset folder.
  • Allow heudiconv datalad option.
• cis-bidsutils
  • Add merging functions just to have them backed up.