Suggestion and offer to help

[stobbsm]

I'm working the UCalgary on the Candig stack (just back from sick leave), and would like to offer some help in containerizing the entire deployment and build process.

Right now, CanDIG relies on locally installed tools and packages, like conda, in order to build containers, manage the deployment, etc.

I'm proposing that everything be done inside containers, discarding the use of conda and local dependencies all together. Nothing being done currently by the build process has to be done outside of a container, and the current process can be quite fragile, and requiring a lot of time to investigate and repair when something breaks.

Please, if you are open to this, it would make things much easier and drop multiple dependencies that currently exist.

[OrdiNeu]

Hello!

Thank you for the offer to help. That sounds interesting, but I wasn't aware that it was possible to build and setup other docker containers from within a different docker container. While we know our deployments can be fragile at times and challenging to debug, we'd like to know more of how we could approach dockerizing the build process before trying it out. Do we know how difficult it would be to adjust our build scripts to run from within another Docker container?

[stobbsm]

Since docker relies on the socket connection to be active in order to run in the first place, it's as simple as adding mounting the socket as a volume: -v /var/run/docker.sock:/var/run/docker.sock:rw,Z, with the ,Z being for selinux based systems to allow a shared context.

This would also remove a strict dependency on docker to build, meaning podman can be used by just passing the podman socket in place of the docker one. It's all compatible, and would give no issues.

This would also eliminate the need to have a full dev toolchain on the main host.

If you are interested, I'd be happy to pull together a proof of concept for a PR.

[OrdiNeu]

We'd still need Docker on the host then, but assuming this still works on everyone's machines this sounds promising. I'm definitely interested.

[stobbsm]

I've put together a proof of concept Dockerfile, although I haven't been able to test it in all scenarios yet. On RHEL based distros, SELinux needs to allow the access, which works under podman, but have had no issues at all under Debian. I've also parameterized the YQ binary installation in the container.

The way it works is to build the container using the Dockerfile, and then run the built container passing in the container socket (podman or docker) and the environment file as volumes.

podman run -it -v /var/run/podman/podman.sock:/var/run/docker.sock:rw,z -v $PWD/environment:/candig/.env:ro builder:latest

It builds and starts the candig stack as expected, no more muddying up the host.

candig-docker-build

More testing will be required of course, but I'm hoping this will harden the build process so it's not so fragile anymore.

[OrdiNeu]

No matter if I run it with/without sudo or using postman/docker, I'm running into an issue with that where the Docker daemon is unreachable:

make[2]: Entering directory '/candig'
docker volume create grafana-data --label candigv2=volume
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
make[2]: *** [Makefile:372: docker-volumes] Error 1

Unsure if there was more to building it other than the build steps outlined in that repo

[stobbsm]

I realized after the fact, sorry. Been a few days since I did it myself, so I forgot, you do need the --privileged flag to podman run to make it work.

I have a bit more work to do until it's stable, and I'm doing that now. Will keep you posted about things.