vatlab / sos

SoS workflow system for daily data analysis
http://vatlab.github.io/sos-docs
BSD 3-Clause "New" or "Revised" License
269 stars 45 forks source link

Singularity to recognize `oras://` #1524

Closed gaow closed 10 months ago

gaow commented 10 months ago

Currently we support automatically select container engine for the following: https://vatlab.github.io/sos-docs/doc/user_guide/h.html#Options-container-and-engine . It would be great if we can add more for singularity. According to:

$ singularity exec -h
Run a command within a container

Usage:
  singularity exec [exec options...] <container> <command>

Description:
  singularity exec supports the following formats:

  *.sif               Singularity Image Format (SIF). Native to Singularity 3.0+

  *.sqsh              SquashFS format.  Native to Singularity 2.4+

  *.img               ext3 format. Native to Singularity versions < 2.4.

  directory/          sandbox format. Directory containing a valid root file
                      system and optionally Singularity meta-data.

  instance://*        A local running instance of a container. (See the instance
                      command group.)

  library://*         A container hosted on a Library (default
                      https://cloud.sylabs.io/library)

  docker://*          A container hosted on Docker Hub

  shub://*            A container hosted on Singularity Hub

  oras://*            A container hosted on a supporting OCI registry

the one we need specifically now is oras:// that we currently dont support.

gaow commented 10 months ago

I should mention that adding engine="singularity" does not work:

[default]
bash: container="oras://ghcr.io/cumc/bioinfo", engine="singularity"
  ls /opt/conda/env/
$ sos run test.sos 
INFO: Running default: 
ERROR: default (id=f8dad9c93b39a550) returns an error.
ERROR: [default]: [0]: 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
script_5849982167480390329 in <module>
----> bash('ls /opt/conda/env/\n', container="oras://ghcr.io/cumc/bioinfo", engine="singularity")

ValueError: singularity engine only allows docker, file, library, and shub container oras specified
BoPeng commented 10 months ago

I should mention that adding engine="singularity" does not work:

So we should at least make the file format open. I think.

BoPeng commented 10 months ago

Please test if the orab branch works.

Edit; the branch should have been named oras, but it does not really matter.

gaow commented 10 months ago

Thank you @BoPeng , this script:

[default]
bash: container="oras://ghcr.io/cumc/bioinfo_apptainer:latest", engine="singularity"
  ls /opt/conda/envs/

works perfectly. I think it is ready to merge to master. Thank you!

BoPeng commented 10 months ago

Merged and released sos 0.24.3.