CLI to get modules installation path

[marcodelapierre]

Hi @vsoch,

yesterday I was starting to think how the container story for the talk in a month might look like; in my mind the central topic is user experience. So, provided the dependencies are available (container runtime, module system, Python), I started to picture myself showing how to get up and running with container modules in the simplest possible way, and ...

pip install singularity-hpc
shpc install vanessa/salad
module use $( shpc QUOKKA )
module load vanessa/salad
salad

Hold on a minute....have I just said QUOKKA ?!

Yes, I have .. because I have realised that QUOKKA would be a nice addition to the SHPC CLI functionalities. As you see, QUOKKA is a way for SHPC to provide the shell with the path where the modulefiles are installed.

We have this for views:

shpc view get <viewname>

So that one can do

module use $( shpc view get <viewname> )

But we don't have this for non view configurations.

My temptation would be to be consistent with views and just introduce shpc get, however this is currently being used to provide the path to the SIF image for a given registry. How should we proceed?

Note that right now one can get the path to a SIF image by means of:

module load <registry>
<registry>-container

So I am tempted to say we could deprecate this old command meaning, and update to the suggestion above. Or we could update the current get CLI to become e.g. get_sif or similar.

Last point: right now the full CLI looks like:

module use $( shpc view get <viewname>)
module use $( shpc QUOKKA )

Should we update it so that the shpc CLI embeds the module use bit? (is it even doable from Python, as module is a shell function?) We can leave as is, just a thought.

What are your thoughts? Thank you!

[vsoch]

Wait I can't tell if QUOKKA is a joke or an actual command? You are saying you want an equivalent of shpc get <modulename> but not for the container? What's a QUOKKA?

[marcodelapierre]

I am glad you asked!!

https://bestlifeonline.com/quokka/

[marcodelapierre]

In this context, it is a command that would basically print the SHPC internal $module_base out to the shell

(fueled by Quokka energy, I forgot to state this in the above description!)

[marcodelapierre]

please refresh your page, I have done some in-comment bug fixes!

[marcodelapierre]

So I am asking for an equivalent of shpc view get <viewname, but not for the views, but rather for the generic $module_base.

[marcodelapierre]

That way it is possible to get shpc set up in a very concise and elegant way, as per my very first code snippet :-)

[marcodelapierre]

(I know, by now you will be lost in quokka cuteness!!!)

[vsoch]

(I know, by now you will be lost in quokka cuteness!!!)

omg, indeed they are adorable!

If this is for a specific module, what about shpc module get <name> ?

[marcodelapierre]

It is for the root dir of the module tree:

$ find $(pwd)
/home/ubuntu/singularity-hpc/modules
/home/ubuntu/singularity-hpc/modules/vanessa
/home/ubuntu/singularity-hpc/modules/vanessa/salad
/home/ubuntu/singularity-hpc/modules/vanessa/salad/.version
/home/ubuntu/singularity-hpc/modules/vanessa/salad/latest
/home/ubuntu/singularity-hpc/modules/vanessa/salad/latest/module.lua
/home/ubuntu/singularity-hpc/modules/python
/home/ubuntu/singularity-hpc/modules/python/3.9.2-slim
/home/ubuntu/singularity-hpc/modules/python/3.9.2-slim/module.lua
/home/ubuntu/singularity-hpc/modules/python/.version

[marcodelapierre]

In this example, we need /home/ubuntu/singularity-hpc/modules, so as to run (in shell or better in .bashrc):

$ module use /home/ubuntu/singularity-hpc/modules

To enable the module tree within the module system.

[marcodelapierre]

The end result being:

$ module avail

------------------------------------ /home/ubuntu/singularity-hpc/modules ------------------------------------
   python/3.9.2-slim/module    vanessa/salad/latest/module

[vsoch]

So you want shpc to do something the module software already does?

[marcodelapierre]

Nope, look at this:

module use $( shpc view get <viewname>)
module use $( shpc QUOKKA )

I want SHPC to be able to spit out the path that module use needs as input.

[vsoch]

We already have:

$ shpc config get module_base
module_base                    /home/vanessa/Desktop/Code/shpc/modules

So you could probably just add a flag that says "only give me the actual value" fairly easily.

[vsoch]

We could even make the single value the default, and then have the extra identifier provided with a flag (that probably is more expected).

[marcodelapierre]

We already have:

$ shpc config get module_base
module_base                    /home/vanessa/Desktop/Code/shpc/modules

So you could probably just add a flag that says "only give me the actual value" fairly easily.

Oh yes good point, thank you Vanessa!

For this specific one, the preferred approach for me would still be to provide SHPC users with a more concise CLI to get the value of this specific config, on the ground that it is used at SHPC setup time.

In alternative, yes, a --value flag that prints values only of the key:values config would also do :-) Or actually, the other way round, as you suggested: value-only by default, and --key-value / -k to print out the key-value pairs. (nested key-values to be handled with care)

[vsoch]

My proposal would be:

$ shpc config get module_base
/home/vanessa/Desktop/Code/shpc/modules

And I don't think we have a good reason to repeat the attribute name. What do you think?

[marcodelapierre]

Quokka approves!

[vsoch]

Would you like me to put in a PR or do you want to take a shot?

[marcodelapierre]

If there is no rush, I should be able to get into this in about a couple of weeks .. wrapping up with my current work projects..finishing off next Friday..! Otherwise feel free to go ahead :-)

[marcodelapierre]

Perfect, I will keep you posted!

[vsoch]

Thank you @marcodelapierre ! Apologies I couldn't jump on it - I'm in full steam work mode. I think I wrote/deleted 5k + lines of code today, I'm shoving down dinner, and am going right back to it.

[marcodelapierre]

No worries, please enjoy some personal time Vanessa!!

I will be cheering for your code hackathon....go Vanessasaurus, lead the team to coding heaven!!