search

khanlab / hippunfold

BIDS App for Hippunfold (automated hippocampal unfolding and subfield segmentation)
https://hippunfold.readthedocs.io
MIT License
47 stars 12 forks source link

Download models inside workflow #247

Closed akhanf closed 11 months ago

akhanf commented 11 months ago

This replaces PR #221, and adds functionality to download models within the workflow. Currently a WIP as doing some testing on it now..

akhanf commented 11 months ago

@jordandekraker looks like the the snakemake http remote doesn't like the dropbox URL for the synthseg models, I'll move them to zenodo (better for consistency and longevity anyhow)..

akhanf commented 11 months ago

OK looks like this is working now, have tested it on graham too. Couple things though:

  1. As before, you need to set HIPPUNFOLD_CACHE_DIR to the folder where you want the models. However, now, if you are running from a hippunfold container, you would need to make sure that env var is set inside the container too, by setting either: SINGULARITYENV_HIPPUNFOLD_CACHE_DIR or APPTAINERENV_HIPPUNFOLD_CACHE_DIR. I think Apptainer is backwards compatible with the use of SINGULARITYENV but it may ultimately be deprecated.. I believe compute canada has migrated completely to apptainer anyways?

  2. If a downloaded model timestamp is earlier than the timestamp on the remote file on zenodo, then it will be re-downloaded..

  3. I haven't been able to confirm how this works on a node without internet (the graham node I ended up testing on seemed to have internet connectivity).. so I think it may be worth testing it a bit more..

  4. It wouldn't be hard to make some versions of the container that have a single model, e.g. for the most popular ones like T1w, T2w, and synthseg.. That could make life easier for people as they wouldn't need to manage a HIPPUNFOLD_CACHE_DIR..

jordandekraker commented 11 months ago

That all sounds good to me, but maybe we should make a note in the documentation (maybe the FAQ) about the default HIPPUNFOLD_CACHE_DIR being in the ~/.cache directory. I don't think its worth building multiple containers with the other models though.

I was also thinking we should try to build this soon since a few people might singularity pull hippunfold:latest hoping to get v1.3.0.

I won't be around much for a few weeks though, sorry, on vacation!