[version 2.0] Make clients separate API modules

[mconcas]

I would like to integrate the Python wrapper around the Singularity executable into my scheduler for launching container for Monte Carlo productions.

Currently I am using Docker with API, but I am very interested in implementing this workflow using Singularity.

I see there are some other command line tools packed with this module.

I was wondering if it would make sense to decouple the pure Python wrapper and make it available also without the other executables (e.g. the shub one, since there are scenarios where it is not necessary)

Is there a common line about this?

[vsoch]

This indeed is a plausible idea! I was trying to keep everything in one module because there is a lot of crosstalk and shared functions. Is there a particular part of the current module that is getting in the way from allowing you to use it? Or are you just interested in a simpler thing? I think an ideal solution would be to have them still served from the same repository, but have multiple installations generated from it. The various versions would then be available via pip. What are your thoughts? One concern that I have is that having multiple modules makes it harder to maintain, especially because there's only one of me :)

[mconcas]

I think that as long as one can keep the actual wrapper separated from the rest of the module, making every command line tool to depending from that, it would be great.

I personally do like the idea to have something more simple to distribute, that maps 1:1 every method and option available in every version of singularity, keeping most backwards-compatible as possible (at least from 2.4 onwards).

Since to the best of my knowledge Singularity does not have the structure of a running service (which is probably by design) it is not expectable to get anything similar to a REST-like APIs. Hence, the only reasonable way to program against it to maintain a lightweight wrapper around the main executable, providing a usable interface, which you have already done.

If you want I can try to highlight the main parts I'm using, while I develop the tool, in order to check whether it is possible to make it a standalone block.

[vsoch]

Actually, Singularity does have services with 2.4, I'm developed a web based experiments container now :)

Yes let's start from there! If you want to highlight the main parts you are using, my proposal is that we wait for when an orchestration tool is needed (akin to docker compose, soon) and then it will be logical to break apart singularity python into singularity client / orchestrator || sregistry / singularity hub. Does that sound reasonable?

[mconcas]

I saw about the singularity instances and example of web service. Cool! I am assuming REST APIs are not planned for Singularity, because of its design (it is not a running service that might be listening to a socket, right?).

For what concerns the orchestrator part it sounds acceptable. So, for the moment, I'm getting the whole package of tools by adopting singularity-python, waiting for a more modular structure for the APIs, fair enough.

[vsoch]

hey @mconcas ! Just to let you know I am (slowly) moving forward with this - the first thing being pulled out is the singularity registry client, and we will go from there!

[mconcas]

Hi @vsoch ! This sounds great! Can't wait to try it!

[vsoch]

hey @mconcas - the registry client is pulled out --> https://sregistry.github.io and following this logic https://github.com/singularityhub/sregistry/issues/93 the core for the singularity client is coming next. Since it's singularity python, akin to singularity registry I will name it (spython). The "singularity" package is being used by the active hub/builders so if it's changed, those functions will need to be put somewhere else first. Anyway, stay tuned now for spython!

[vsoch]

Okay, spython should be good for use (and more testing if you would like to contribute!).

[mconcas]

Thanks @vsoch, I'll have a look as soon as I can!

[vsoch]

This is totally done, all clients are removed.