Cylc as a containerized service

[hjoliver]

@kinow - your excellent "distributed" example is great for testing cylc in various ways (remote hosts with configurable network latency - brilliant!) but I'm wondering how a containerized cylc might work transparently for users.

Currently we expect multiple cylc versions to be installed side by side on suite and job hosts, and we have a built-in way to ensure that suite jobs invoke the same cylc version (if available) as their parent suite. But of course Python etc. must be compatible on the systems, and cylc has no control over that.

I wonder if we could have containerized suite server programs (so the containers, like the suite server programs themselves, would live for the duration of the suite run) which would submit jobs that invoke similarly containerized cylc versions for task message commands etc.?

[kinow]

Currently we expect multiple cylc versions to be installed side by side on suite and job hosts, and we have a built-in way to ensure that suite jobs invoke the same cylc version (if available) as their parent suite. But of course Python etc. must be compatible on the systems, and cylc has no control over that.

This sounds like an excellent use case for containers.

I wonder if we could have containerized suite server programs (so the containers, like the suite server programs themselves, would live for the duration of the suite run) which would submit jobs that invoke similarly containerized cylc versions for task message commands etc.?

I think it makes sense. Jenkins nowadays is being used to schedule jobs, which will spin up whole environments with clusters of containers running MySQL, Postgres, PHP, Java, Python, Celery, Redis, etc, that are later destroyed or kept for troubleshooting. There is normally one point of truth that aggregates the results (in Jenkins it is normally a Jenkins master, which servers the Web UI) and displays them back to users.

This use case is quite similar to what you described above I think. In the past we used to have to install different versions of PHP, Python, NodeJS, etc, in each slave. And maintaining all these different versions and making sure each job was using the right tools/versions was a huge challenge. Now we just create a whole environment (which re-uses bits and pieces through Docker images & layers).

We are using it successfully both for running tests in Jenkins, but also some of our new production services are being (or will be in a few weeks I think? Operational issues...) served directly from Docker containers :tada:

If I remember well, there was a presentation somewhere saying that our new supercomputer would be able to even run Docker containers? Though I may be wrong. However, I know some tech-savvy researchers are using containers, and looks like even an intern used it recently to run a deep learning program (which sounds really interesting and I will chase to see if I can learn more about it).

If you are keen to experiment with it, I think if the whole journey is documented, it would become an interesting paper/article/blog post :-) (which I'd really like to read!)

[kinow]

Oh! And thanks for the excellent! I wasn't sure if others would find it useful. It took me some extra hours of reading through the best practices to make it more presentable, but now at least my next Dockerfiles will look a bit better. :nerd_face:

[hjoliver]

I wasn't sure if others would find it useful.

I guess it strikes me as useful mainly in suggesting possible uses of containerization in packaging and/or testing cylc in the future (as I haven't dived into Docker before, somewhat embarrassingly).

[kinow]

That's normal. I believe you skipped most of the hype phase. It's getting less hype-y, and easier to keep up with the changes.

Besides re-using the operating system's kernel, initially I believe they also used the same networking layer. But from what I recall, they decided to completely rewrite the network layer. There were also breaking changes between versions, regressions.

So now we have better docs, larger community, and more use cases to use as basis :-)

[hjoliver]

there was a presentation somewhere saying that our new supercomputer would be able to even run Docker containers?

I'll see if I can find out.

I've also heard that Docker security issues (maybe just daemon-as-root, and root in the container?) make Singularity preferred in HPC https://www.sylabs.io/

[kinow]

I've also heard that Docker security issues (maybe just daemon-as-root, and root in the container?)

There are heaps of containers defaulting to the root user (as the ones created by the dockerfile in this repository, cough-cough). But so far the attacks/issues in containers that I have heard about, were not unheard in VM's/boxes (e.g. docker privilege escalation & vmware privilege escalation).

make Singularity preferred in HPC https://www.sylabs.io/

Hadn't heard about it. I have no strong preference for container solutions. But I normally use Docker just because it's the one I am more familiar with, but now will keep an eye for Singularity and try to read about it later. Thanks!!!

ps: privilege escalation on Singularity: https://www.sylabs.io/2018/05/whatsnew-singularity-2-5-why-affects-everyone-using-containers/ from May this year. Guess CVE's happens to all tools at some point in time

[hjoliver]

There was a talk at Sci Co Rotorua that mentioned Singularity. As I recall, it requires root to make a container, but unlike Docker there is no persistent daemon running as root (I think).

[kinow]

Oh, that sounds interesting. I think it would increase the security, and wonder if that'd be possible in Docker too - maybe someday.