Pushing singularity image to singularity-hub

[justinblaber]

Is there a way to create a singularity hub repo (similar to docker-hub) to push singularity images to? Instead of using a github + webhook to rebuild automatically. With dockerhub we can just push images whenever we feel like it with docker push and then assign whatever tag we want.

I guess my main issue here is if we have a large amount of data that we package in the singularity container what exactly would be the best way to go about getting it inside the container with a Singularity file and a github webhook? Would have to host the data some where and do a wget with the Singularity file or something?

What we're currently doing is creating docker images, adding the data in with ADD, hosting them on dockerhub, then converting them to singularity image files. It would be great to then push these onto singularity hub somehow.

Thanks,

-Justin

[vsoch]

You should deploy a Singularity registry for your institution, and you can push to that!

[vsoch]

https://singularityhub.github.io/sregistry

[justinblaber]

So just to be clear, singularity registry is like dockerhub but we would have to host it?

Whereas singularity-hub is a github dependent webhook based dockerhub basically?

And if we want the functionality of doing a singularity push <singularity image> we would definitely need a singularity registry?

I'm just trying to get the thousand foot view of our options here.

Thanks!

-Justin

[vsoch]

Hey @justinblaber !

Now that I have a few moments I can take the time to give you a more thorough response! Singularity Hub is a cloud hosted (Google Cloud) builder service, meaning that there is a central server and workers (I call affectionately "the builders") that are deployed on demand to build an image. Singularity Hub is hosted by Stanford, so this means two things. First, while I would wish to be able to provide unlimited builds of things for all times, we have to be reasonable with respect to the number and content of containers. This is the rationale for having the build controlled, and not arbitrary upload of images that you have built. If I allowed for push (and increase in number and size, generally) I would have to think about a long term sustenance (pricing model) which I don't want to do. second is security. Having custom tokens / authentication and an ability to push to a server, period, opens up a lot of loopholes that I don't want to do. At least through Github the flow is more tightly controlled, and in a way GIthub acts as the middleman for handling user authentication and permissions. Finally, we can think about the intended audience. The intended audience for Singularity Hub is the small scientist that may not even have a resource to build on. But I bet you he has Github and can upload a file there! And then share his reproducible container. Since I want Singularity Hub to be free for this use caes (that I hope it serves now), it has been designed in this way.

Now Singularity Registry! Sregistry was designed out of the need for the opposite thing - having a shared resource or institution that wants more scaled building, and control of it. The idea of Singularity Registry is that while one central "Singularity Hub" couldn't be the sole carrier of all containers (who is going to pay for it?) an instutiton can be empowered to serve (and control access to) their own images, and push/pull freely. Singularity Registry is like the open source, community developed version of Singularity Hub. To help with many different deployments / use cases, the different features are added as plugins. If your need isn't met, we can very likely create a plugin for it. My hope is that institutions, by way of SRegistry, can deploy their own and then we have more of a distributed model for sharing images.

Finally - future things! I know that Sylabs (the company that will be behind open source Singularity) has plans to offer a paid service sort of deal. In my mind, this is a different goal because it doesn't fit the original use case that I had in mind, but it would be an option if you can afford it.

So as a developer and someone who cares a lot about open source tools, I would advocate for you to look into the second option, and start a discussion about your needs and any missing things. Then we can do like we always do, and opensourceitup! :)

[vsoch]

We just merged teams into sregistry this morning, and next on the queue is security scanner (likely clair) and custom builders, if you have thoughts on those things!

[justinblaber]

Thank you so much for the info!!! I'll try out Sregistry and see how it goes.