Feature request: Do not create volumes on request

[tal-zvon]

I don't know if this is possible. This may be a docker limitation.

This is my use case:

I have a Django project stored as a git repo on a self-hosted GitLab server. I'm setting up a CI/CD pipeline so that GitLab runs tests on the Django project on every commit. This will be done using a docker runner.

My Django project depends on mariadb, so my .gitlab-ci.yml file starts the mariadb container as a service. The issue I'm seeing is that every time this happens, the mariadb container creates a new docker volume. When the test is over, and the mariadb container is shut down, the volume remains.

Eventually, I'm going to end up with hundreds, if not thousands of these volumes - one for every test that GitLab runs (on every commit to my Django project).

There are, of course, other solutions, like:

• Having django use sqlite when running in a container (fairly easy to do using env variables)
• Having a cron job on the GitLab runner machine that prunes unused docker volumes every once in a while

It would be nice however if instead of these hacky solutions, we could just set an env variable that told the mariadb docker container not to create a new volume as it starts up.

Is this at all possible?

I feel like my use case isn't all that unique - plenty of people would want mariadb to be started automatically as a docker container using a GitLab CI/CD pipeline.

[grooverdan]

Quick thought is that the CI/CD should be creating and destroying a volume in the pipeline and passing this explicitly to the container. There is VOLUME in the container that implies some behaviour.

[grooverdan]

Good to see you followed up with Stack Overflow.

[tal-zvon]

I actually posted there first.

I'm pretty new to GitLab CI/CD, and CI/CD in general, but as far as I can tell, the only file that configures this in GitLab is the .gitlab-ci.yml file. You can use this file to specify commands that should be run within the docker container, and you can specify environment variables that should be exposed to the containers, but you can't tell it to configure the docker containers it starts up in a certain way. You also can't run any kind of shell commands on the docker host (the GitLab runner machine). Therefore, you can't tell it to create a docker volume, pass it to the container, and destroy it afterwards.

I was hoping you guys would have either come across a similar situation, had some suggestions, or could think of a way to do this from within the container when a particular environment variable is set, that could be added as a feature.

Perhaps the GitLab CI/CD system is too new to support this, and will eventually allow you to specify docker container settings. It doesn't sound like it's possible to do anything from within the container.

[ottok]

If Docker volumes are not explicitly named, they will get random names, and if not explicitly purged, they will stay around. That is kind of the purpose of data volumes in Docker.

Here are typical commands I use in various Docker environments to clean up old images/volumes etc:

docker volume prune --force && docker system prune --force # system docker-compose down -v --remove-orphans # compose project

Or use --rm when starting new containers.

Gitlab-CI is very mature and works well. If you end up in an un-optimal solution then you need to check that your solution is correctly architectured and that you've read the man pages of the commands you use. I hope this gets you on the right path.

[tal-zvon]

If you end up in an un-optimal solution then you need to check that your solution is correctly architectured

That is literally the purpose of this post, as well as the stack overflow post.

I appreciate your response, but your "solutions" are very high-level, and have no way to be implemented. You're basically saying "identify the problem, and solve it". Your suggested shell commands only work when you run them on the docker host, which as far as I can tell, GitLab-CI can't do.

The only solution I've found is to add this cron job to the GitLab runner:

* * * * * docker volume prune -f

or the slightly smarter:

* * * * * bash -c 'systemctl is-active --quiet docker && docker volume prune -f'

This solution essentially disables docker's permanent storage of volumes that are no longer in use by a container - anyone who wants to use this should be aware of this, or it can unexpectedly delete your data.

I was aware of this "solution" even before opening this issue - I refer to it in the original post of the issue. This works, but as you put it, this is "un-optimal".

Sounds like there's no way to fix this from within mariadb-docker, so I'm closing this issue.

[grooverdan]

Or when gitlab-ci containers are removed docker/podman rm the docker rm -v option removes anonymous volumes.

But this looks like something GitLab-CI should be doing already.