[MESSAGE] Singularity Hub Temporary Taken Down

[vsoch]

We've identified a malicious user, and have taken Singularity Hub down until further notice. All containers and collections are safe, and this post (and Twitter) will be updated with further information.

[kapsakcj]

I've subscribed to this issue for updates, but where on twitter? Does singularity-hub have it's own twitter account?

[vsoch]

On Twitter it would be me, @vsoch. But I'll be posting updates here, you can count on it.

[dhirsch8]

Any estimates for when Singularity hub will be running again? Thanks!

[vsoch]

The latest update is that folks on my team that need to give feedback are out on vacation until the end of the week. And as I previously said, I'll update this issue when I know more.

[vsoch]

Update: I'm happy to report that we have finished investigation, and have a plan in place to move forward! I'll be testing the server this weekend, and hoping to have it live again sometime next week. I'll give you a thorough review of new policy and changes at that time. 🎉

[vsoch]

Singularity Hub is open for business! Usage will be highly limited to the end of the week, and then resume normal usage next week. See the limits and other notes -> https://singularityhub.github.io/singularityhub-docs/2019/release-announcement/

[tobsecret]

I'm a teeny bit confused about this bit of documentation: https://singularityhub.github.io/singularityhub-docs/docs/interact So I should or shouldn't use singularity pull? Or I should only use singularity pull after using singularity shell? I'm using shub for images used in my nextflow pipelines, which I think uses singularity pull. Should I convince the nextflow folks to instead use singularity shell and then singularity pull?

[vsoch]

You should definitely pull! To be clear - when you pull, you are doing a GET request to get a container binary in the present working directory (or cache). Then once you have the container, you can exec / shell / run to your heart's content. What you don't want to do is this a million times:

singularity run shub://container

If you are running something on a parallel resource, obviously you also wouldn't want to pull a gazillion times. So - pull isn't universally good/bad, it all depends how you use the commands.

[vsoch]

For your nextflow pipeline, you'll want to edit it so that you pull before running it, and then issue commands to the container binary. You definitely don't want nextflow pulling the same containers a bunch of times, it will count toward the quota.

[tobsecret]

Aah, okidoki, got it! Thanks for clearing that up! Nextflow by default pulls once before pipeline execution and then interacts with the image locally. I think I just misunderstood that documentation bit - thought singularity shell remotely and then using singularity pull from within the shell would not use a GET request for the weekly limit but that doesn't make sense :sweat_smile:

[vsoch]

Exactly! Just be careful if you are running a nextflow pipeline across a huge number of nodes, because if all the nodes don't have the container cached they will all start the download (and each download counts as a request.)

[hariszaf]

Hi! I would like to ask you whether the building options and especially the building through GitHub (using a file which needs to be called "Singularity") will be up and running until next week as well. Thank you very much for your time! Haris

[vsoch]

hey @hariszaf ! Yes, I intend to (given that all goes well) open up building generally for all users likely end of this week. In the meantime, I've added you as a testing user if you'd like to give it a go. And you are allowed to add an extension to a Singularity file, and have it be anywhere in the repository (e.g. subfolder/Singularity.tag). See the docs here https://singularityhub.github.io/singularityhub-docs/docs/getting-started/recipes#multiple-recipes for a more detailed examle.

[hariszaf]

Vanessa thank you very much! I am going to test is as soon as possible! That is really a great feature of Singularity Github! ;)

[vsoch]

Your builder is working on it right now! I'll keep an eye on it. Remember that the interface no longer refreshes for you, so you'll need to refresh the page manually for updates.

[vsoch]

Just for fun, my builder gives them fun (interim) names, and yours is dinosaur-nunchucks-1739.sif Don't worry, the final thing will be named based on the image hash!

[vsoch]

hey @hariszaf just a heads up that your build finished, but there is a permissions error coming from the builder. It would have notified you a few minutes ago, but the message isn't getting through. I'm going to need to debug this manually, so please stay tuned (and don't delete / trigger anything for now!)

[hariszaf]

Οk @vsoch !! If I can do anything please let me know! I am not that experienced with Singularity, but this recipe used to be just fine before the attack - that's all I can say! dinosaur please do not go extinct! :laughing:

[vsoch]

Okay, so it looks like you deleted a container while it was in the process of being built, so the message is correct. The builder could not return the response because the container in question (with id 10245) doesn't exist. Can you share with me the steps you took to do this?

[vsoch]

Indeed, the recipe does build fine, but it didn't have a container to return to :)

[vsoch]

The current container you have is 10246, so it looks like you might have deleted a container?

[hariszaf]

Yes! Before adding an extension to the Singularity file on GitHub, I removed the previous container I had on Singularity Hub. I should not had done that, I suppose..

Here is my recipe: https://github.com/hariszaf/pema/blob/master/Singularity.v1

[vsoch]

Yes, what actually happened is your first recipe had already started a build, you deleted the container it would return to (note you don't normally need to do that, the container will be replaced with the new one unless you freeze it) and then there was no container to return to.

Give me a second and I'll ping the pending container for you. Please don't push again or delete anything!

[vsoch]

lol, okay looks like you did anyway! Please don't do any deletes / pushes until this builder finishes!

[hariszaf]

I got your point! Sorry for the trouble! I believe I understood a bit more what you're doing! :) The status now is "running" on my screen!

[vsoch]

Yep, and the container id is correct, so this one should be ok :)

[hariszaf]

Vanessa in any case, many thanks for all your time and help! :)

[vsoch]

okay @hariszaf, refresh the page, you're good and green!

[hariszaf]

i can see i am green!! :D

i am about to test if it works fine and let you know - just as a confirmation!

[vsoch]

I'm watching you pull - don't forget you need to specify the tag (without one it defaults to latest, and you'll get 404).

[vsoch]

Try:

$ singularity pull shub://hariszaf/pema:v1

[hariszaf]

Yes yes! I am pulling it now.

[hariszaf]

@vsoch everything's doing great! hope you ll never need to deal with such hard times in the future! thanks again for all your help! Haris