Question about usage of named Docker volumes

[benibr]

The current usage of the Docker volumes is:

• Data is stored in directories inside the repo (state, mig, certs, etc.) and not within Dockers own structure (/var/lib/docker/volumes)
• These directories might get removed by make targets like clean or distclean
• Named Docker volumes are created via docker compose, with the above mentioned directories as targets
• Those named volumes are then assigned to containers
• The named volumes are never cleaned up by any make targets

In some corner cases this leads to unwanted behaviour:

• If the target path of a volume should change, the volume is not recreated automatically, thus after rebuilding everything the containers run with the old volume
• Same happens if the path of the docker-migrid repository changes
• In both cases the wrong path is not visible in the docker-migrid repo, only if one inspects the existing volumes
• Deleting the named volumes does not delete the data, unlike one might expect

I would like to the discuss the option of completely removing the named volumes and setting the bindmounts only in the volumes list of each container. This would result in

• no more hidden leftovers after make distclean
• no more manual cleanup necessary if directories are changed
• possibility to see what is mounted whereto in the compose section of each container
• removal of 400 lines in docker-migrid, resulting in better readability

[benibr]

For clearification, an example of the change would look like this:

Before:

services:
  migrid:
    ...
    volumes:
      - type: volume
        source: httpd
        target: /etc/httpd/
volumes:
  httpd:
    driver: local
    driver_opts:
      type: none
      device: ${DOCKER_MIGRID_ROOT}/httpd
      o: bind

After:

services:
  migrid:
  ...
  volumes:
    - ${DOCKER_MIGRID_ROOT}/httpd:/etc/httpd/

I already have I prepared change, if there's interest, I can open a PR.

[jonasbardino]

It's a good point and I agree that it's currently a bit of a mess. This is definitely one of the areas where we need your involvement and to converge on a suitable solution for all parties involved - in line with a discussion on the recent meet-up :)

I know @Martin-Rehr did some work on the integration of Lustre mounts at a test site and I suppose AU and you might have something similar in place for your sites.

If your work here already takes that into account or simply applies transparently with network storage, I think we can proceed. Perhaps waiting with the final merge until Martin is back from vacation.

[rasmunk]

Just for the record, the reason it got implemented that way by me and patchofscotland way back when, was because we did active development directly in the docker-migrid repo. Here we used the current bind mount to ensure that changes made in for instance the ./mig path on the host machine would be reflected and take precedence over what is inside migrid container. Since we have moved way past that setup at this point, the reason for the old bind structure also goes away. At least in my mind:)

[benibr]

That makes sense and I think we should keep it for development purposes this way. However for production it might be a good idea at some point to use only the migrid version from the container image. Anyway that's something to discuss for another time.

[benibr]

Unfortunately I have to resign here. My suggested solution will implicitly create a bind mount instead of a Docker volume which works in case the application inside the container populates the data inside the mounted directories (mysql does that for example). With migrid instead we're relying on the data population mechanism from Docker which apparently works only for volumes not bindmounts. https://docs.docker.com/get-started/06_bind_mounts/#quick-volume-type-comparisons

So I will open a PR for the removal of the volume in make distclean to avoid the problem described in the first issue text.