search

ReproNim / containers

Containers "distribution" for reproducible neuroimaging
Apache License 2.0
26 stars 16 forks source link

Rewrite create_singularities in Python #73

Closed yarikoptic closed 2 years ago

yarikoptic commented 2 years ago

The goal is to rewrite bash https://github.com/ReproNim/containers/blob/master/scripts/create_singularities in Python.

Rationale:

While working on it, keep in mind upcoming extensions such as #74

jwodder commented 2 years ago

@yarikoptic

if written as async, enjoy parallel execution/creation of containers

Exactly what actions do you envision being run in parallel? Would it include the datalad operations (saving etc.)?

get rid of unused stuff: there is no longer singularity hub and its API so that code could be gone

I don't seem to see anything in the script for interacting with the singularity hub; exactly which parts are currently unused?

yarikoptic commented 2 years ago

Exactly what actions do you envision being run in parallel? Would it include the datalad operations (saving etc.)?

I thought just about creating them (singularity build). DataLad operations probably should not be async unless they all go through the same shared git annex add --batch process or smth like that. But the easiest would be to just lock datalad operations and thus force them going sequentially. But since we run this script on cron, there is no "urgency to interact", if you think such async would be more cumbersome than useful - ok to not bother.

get rid of unused stuff: there is no longer singularity hub and its API so that code could be gone

I don't seem to see anything in the script for interacting with the singularity hub; exactly which parts are currently unused?

it seems indeed that I already removed that functionality in 76997d626a827244dd778eae06fb659bee42835d etc. So, never-mind that part ;)

jwodder commented 2 years ago

@yarikoptic For the "upcoming extension" #74, exactly what would be done for that? Is it just converting more Docker images to Singularity or something else?

yarikoptic commented 2 years ago

yes, I think it would be pretty much the same conversion and just with a different source for getting the listing of those containers. So may be nothing specific is to be done for that ATM.