GCP buckets as registry

[evalanguaria]

Is your feature request related to a problem? Please describe.

Describe the solution you'd like We would like to use GCP buckets as a registry for our containers and use those with shpc and I believe currently that is not possible. It would be really helpful for us if that capability was added. Thank you for your consideration. Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

[vsoch]

We can definitely think about this! To clarify - you mean to install to a system from a bucket (and still have the final container be stored on the local filesystem) ?

[evalanguaria]

To clarify - you mean to install to a system from a bucket (and still have the final container be stored on the local filesystem) ?

Thank you for looking in to this. Yes, that is one of the use cases we had in mind. Could there also be a way to have the modules and containers in a remote repository (such as a google bucket) and load them from there?

[vsoch]

Could there also be a way to have the modules and containers in a remote repository (such as a google bucket) and load them from there?

If you can engage with the LMOD or environment modules developers and they have ways to do this, it could be supported here too! Since users of HPC are utimately going to run the software on HPC, at the end of the day something needs to be pulled/installed locally. But if you have something different in mind, could you share with me and we can think it through? If you are wanting to install these to 'some other places aside from HPC" and thus HPC modules aren't the best deployment vehicle, we could extend shpc to allow some other type.

[evalanguaria]

Our system is not exactly an HPC but we felt modules could help us organize better our software and tools we provide to users. What we have is a kind of sandbox environment in GCP that has no access to internet with each user able to spawn a virtual machine in their sandbox environment (a GCE image). So what we are looking for is a way to enable users to use software they need (like different versions of Python and R, and associated packages) but we would like the virtual machine image to be as small as possible containing only the absolutely required software(such as Lmod, shpc) so that the time it takes us to build the virtual machine image is shorter and then to enable users to add software they need to their own home disks. And having these modules in a bucket(that are mounted to the users’ directories) where users could pull from to their own home disks could make it possible for us not to have everything in the virtual machine image. If my explanation is not clear or if there are further things you would like to discuss please do let me know. Thank you again for your time.

[vsoch]

oh I love this idea! I have a personal account on Google Cloud so I can definitely work on it - let me think about a few designs and get back to you!

[evalanguaria]

Great! Look forward to your ideas. Thank you.

[vsoch]

hey! Sorry for the delay here - I'm going to move this up in my personal project TODO so I should be able to test out some things soon.

[vsoch]

@evalanguaria here are a few ideas to start discussion:

1. You could have a central provider of modules that any user could install from. This would mean selecting the container images/recipes you want, and then mirroring them on your internal OCI registry (e.g., https://cloud.google.com/container-registry). This would mean the automation (somewhere) would need to still get updated recipes, and then pull / push the containers to where the user can access them, and then update some internal repo that has a custom recipe that points to your versions.

2. Have more of a builder service that pre-generates an environment with container modules based on a user request. This would require internet connectivity, but then you would save the image/environment for use as a GCE image.

3. Have a service that pre-installs all new modules to somewhere (e.g., meaning it pulls the Singularity container or container of choice and creates the module) and then figure out a way to have that available.

Do any of those sounds in a direction that is interesting? :thinking:

[evalanguaria]

@vsoch , thank you for working on this. I think options 1 and 3 would work for our use case. We would like to have the modules available in some place that users have access to (we were experimenting with bucket/nfs mounted to users disk) and users can then install the modules they require.

[vsoch]

Cool! So I think 3 would be more reproducible and easy to maintain, so I'll start thinking about this one. This is something I need to do in free time / off hours so I'll have more time in evenings and weekends (but soon).

[evalanguaria]

Thanks a lot.

[vsoch]

@evalanguaria so would a script that takes some set of modules and then generates them in Google Storage satisfy the need? It seems fairly simple - we'd likely just run a VM mounted to a bucket, and then a bunch of shpc install commands. Let me know if your thinking has changed since we chat.

[evalanguaria]

@vsoch one thing we have found out during our tests is that GCP buckets are very slow when executing any of the module commands (spider, avail, load etc) and gcp filestore were much faster so we would instead like to use those. Other than this our requirements haven't changed. So if the script could accept a filestore or bucket address for storing the generated modules, that would be great. Thank you.

[vsoch]

Can filestore easily retrieve data from a Google bucket? That would be easy for me to test and develop - I'm not sure I could spin up and pay for a 1TB filestore instance! :laughing:

Oh yeah I think maybe yes? https://cloud.google.com/filestore/docs/copying-data

[evalanguaria]

:) Yeah, filestore is expensive and yes it is possible to copy between gcp buckets and filestore.

[vsoch]

@evalanguaria sorry more questions. So to clarify some terms - a registry is just yaml files. The yaml files have metadata to pull containers. But you want in Google Storage to have the containers and modules - so it's less of a registry and more of a cache. Is there any reason you can't just run a VM, install the modules you want there, and then save them to your storage? Sorry if I'm missing something here.

[vsoch]

@evalanguaria here is a script that does what you desire (I think!) https://github.com/singularityhub/singularity-hpc/pull/638

You can further automate that - e.g., if you put your list of containers (the containers.txt file) and the script in a storage root, you could create a VM with a start command to automatically install the dependencies and run the script against your desired file.

[evalanguaria]

@vsoch my apologies for the late reply. What you suggested in ur question is one way (or possibly the only way?) to go about this (which we have been doing during our tests) where someone such as a system admin would do this and the modules would then be available for the users to use. But what we wanted at first was for the users to be able to install the modules they need themselves in the sandbox environment where there is no access to the internet and hence the need to keep the yaml and container files in some google storage that they have access to like a bucket, filestore or container/artifact registry. In shpc only github and local paths can be given as a registry but we wanted to use buckets or artifact registry for that, then because there is no internet in users VM we wanted also the containers to be available in such places that they have acess to. I hope this answers your question. Thanks alot for the script, i haven't gone through it yet but i will as soon as i can.

[vsoch]

Yes! And I think(?) I implemented what you were imagining - it creates a cache of some set of containers / modules you want, and you could then mount somewhere without internet.

[vsoch]

@evalanguaria have you had a chance to try out the script? If so, we can merge and close the issue here. If not, let me know!