Add Instance control to Singularity Python

[wkpalan]

The following command is what I use for starting an instance and running a command, especially because I am using MySql service.

singularity instance.start \ 
    --bind $PWD/GOMAP-data/mysql/lib:/var/lib/mysql \
    --bind $PWD/GOMAP-data/mysql/log:/var/log/mysql \ 
    --bind $PWD/GOMAP-container:/opt/GOMAP \ 
    --bind $PWD/GOMAP-data:/opt/GOMAP/data \ 
    --bind $PWD:/workdir \ 
    -W $PWD/tmp \ 
    $img_loc $instance_name && \ 
singularity run \ instance://$instance_name $@
singularity instance.stop $instance_name

Would it be possible to add the instance.* commands to Singularity Python so that starting and stopping instances with proper bindings could be called from python. This allows for reading new config for binding directories from json or yaml file.

[vsoch]

Per discussion in gitter, yes I definitely think so! I'll get started on this soon and hopefully have something for you to test by the weekend.

[vsoch]

Some ideas for what I think would make sense - the user would want to create an instance:

from spython.main import Instance

myinstance = Instance(name="avocado", binds=["..."])
myinstance.start()
myinstance.logs()
myinstance.start()

> print(myinstance)
instance://avocado

> myinstance.name
instance://avocado

or start on definition (which should be default)?:

myinstance = Instance(name="avocado", binds=["..."], start=True)

And I think it would make sense to have control of the instance be from the instance itself, e.g.,

result = myinstance.run()

As opposed to:

from spython.main import Client
Client.run(myinstance)

And I think it would make sense (if possible) to have the Instance give easy access to show status, logs, etc. Will test this out! :)

[wkpalan]

I agree. I think having control functions on the instance object itself makes more sense opposed to importing the spython client and running the instance.

This way multiple instances can be started and followed through within the same script as variables opposed to saving the name in a separate list.

I'll let you know if I think of anything else that might make sense.

[vsoch]

@wkpalan I still need to write tests (and run them) but there is a PDF attached here if you want to see how things will work! --> https://github.com/singularityhub/singularity-cli/pull/43

I'll try to make some more time this weekend for those tests, and I'll let you know when it's ready for your testing!

[wkpalan]

Thanks for the heads up.

Sent from my Android

On Fri, Jun 29, 2018, 9:33 PM Vanessa Sochat notifications@github.com wrote:

@wkpalan https://github.com/wkpalan I still need to write tests (and run them) but there is a PDF attached here if you want to see how things will work! --> #43 https://github.com/singularityhub/singularity-cli/pull/43

I'll try to make some more time this weekend for those tests, and I'll let you know when it's ready for your testing!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/singularityhub/singularity-cli/issues/42#issuecomment-401511566, or mute the thread https://github.com/notifications/unsubscribe-auth/AGDytp7HvtoTNrIFoG7yAAfkE9yCLg0Hks5uBuOOgaJpZM4U6ZBi .

[wkpalan]

Sorry it took this long to test it, but I started working on using a python script for running apps within the container on hpc. Looks like my instance starts but hangs after it starts in python. I cannot get to the next line to access the instance that was started

from spython.main import Client
Client.pull(image="shub://jekriske/lolcow:lowfat",name="lolcow.simg")
myinstance = Client.instance('lolcow.simg',name="lolcow")

[vsoch]

I think you are likely pulling to a SINGULARITY_CACHEDIR that is in your home (that is filled up on the resource). Try exporting it to be somewhere with lots of room:

export SINGULARITY_CACHEDIR=$SCRATCH/.singularity

and trying again!

[wkpalan]

It looks like even after that it hangs. Let me see if it improves after I install the latest version from master

[vsoch]

Note that this was also tested (and developed) on singularity 2.5.0. If it doesnt work again please provide:

• the version of singularity and environment
• an example (and output / result) of running the commands on the local host
• the same commands run with spython, and the result

This will let me debug the issue. Thanks!

[vsoch]

As per our previous discussions, please also provide the complete commands that you are using and the output, because the bug might be there too :wink:

[wkpalan]

Singularity version - 2.5.2-feature-squashbuild-secbuild-2.5.0.bad17afe

localhost commands

singularity pull -n lolcow.simg shub://jekriske/lolcow:lowfat
singularity instance.start lolcow.simg lolcow
singularity run instance://lolcow

Python Script

from spython.main import Client
Client.pull(image="shub://jekriske/lolcow:lowfat",name="lolcow.simg")
Client.instance(image="lolcow.simg",name="lolcow")
Client.run()

[vsoch]

You aren't running the instance - you are creating a client, then creating an instance (but not getting the return uri from it) and then running the client (that doesn't have anything associated.) Again, don't forget the return argument! Remember, there are guides that walk you through this. --> https://singularityhub.github.io/singularity-cli/commands-instances

For example, from reading, I would try this....

from spython.main import Client
myinstance = Client.instance(image="shub://jekriske/lolcow:lowfat",name="lolcow")
print(myinstance)
result = Client.run(myinstance)

[wkpalan]

I ran the same exact commands and have been waiting for 5 min, but hasn't gotten to the print statement.

[vsoch]

As I asked above, please run the command with LOCAL singularity and send me the output first. I think you might need to read things / instructions / docs better :) Is english your native languag?

[vsoch]

Thanks for the fixes btw!

[wkpalan]

I have also figured out what was causing the instance control problem in my machine. I have tried it in three different Ubuntu machines and it always hung so I added a couple of lines to fix my current instance issue.

[wkpalan]

English is not my native language, but that doesn't seem to have impacted my ability to copy paste the lines from the comment above.

[vsoch]

ok no worries! Just for fyi, I've been working this evening to add automated builds (and testing of them) to the repos here so we can be sure the Docker containers build too :)