Future release workflow of docker-migrid

[benibr]

Godmorgen :-) I want to suggest a change to the commit/release workflow of the docker-migrid project.

Currently the repository contains multiple examples, some for development and some for real deployments etc. At KU we deploy our custom docker-compose and env file. AU uses a custom env and a overlay file is used for docker-compose which superseeds some setting from the erda config. In the end the env is always deployment specific.

I suggest that we make the following changes for the future:

1. remove all deployment specific configuration (like dev.erda.dk etc) files from the repository
   1. the docker-migrid repo should be as clean and generic as possible
   2. all site specific deployment config should be part of the deployment process (eg. Ansible)
2. keep 3 example deployments:
   1. one for development (migrid.test with local DNS etc.)
   2. one for SIF development (like the one above but with GDP enabled)
   3. one as advanced production example that serves as baseline for AU overlay (migrid.example.com, without local DNS etc.)
3. remove the GIT_REV from the example env files and only use GIT_BRANCH
   1. then no more version bump is necessary when migrid-sync gets updated
   2. meaning that the docker-migrid repo only changes if there are change within repo itself
   3. a production setup has it's own env file anyway and one can set the GIT_REV there for version pinning
4. we can then introduce a semantic versioning to docker-migrid following https://semver.org/
   1. this will make it way more easy for admins to decide when to use a new version of docker-migrid

Feedback appreciated :-)

[benibr]

@jonasbardino @Martin-Rehr If you do not object would start to work on a draft.

[benibr]

I see the following

Todos for KU:

• prepare PR for the decided changes
  • [x] remove site specific files
  • [x] keep a advanced production example is usable by AU overlay
  • [x] align deployment names with filenames
  • [x] remove GIT_REV from all configs
  • [x] add version tags to docker images

Todos for AU:

• test and review the provided PR

Todos for UCPHHPC:

• Ensure the deployment configs for erda etc. are stored and used outside the docker-migrid repo

[jonasbardino]

We can surely move the advanced_X.env files like dev-erda, dev-sif and bench-erda to e.g. a new docker-migrid-erda repo or an examples subfolder if that's cleaner. Then we just need to coordinate explicitly about pushing any future mandatory vars to KU and AU envs.

[jonasbardino]

Removing the explicit revision from the confs is fine with me, but you probably still want to pin versions at least in production. We also still generally need explicit revision info in relation to support and reported issues. Perhaps we should print it during container launch? (available in /home/mig/active-migrid-version.txt)

[benibr]

We can surely move the advanced_X.env files like dev-erda, dev-sif and bench-erda to e.g. a new docker-migrid-erda repo or an examples subfolder if that's cleaner. Then we just need to coordinate explicitly about pushing any future mandatory vars to KU and AU envs.

I'd say we'll have the two main examples defaults.env as a development environment and the advanced.env as a production environment. A new value will be added to both of them, so an admin can see those changes and adjust her environment accordingly. Later we can also think about change the major version if a breaking change is introduced.

[benibr]

Removing the explicit revision from the confs is fine with me, but you probably still want to pin versions at least in production. We also still generally need explicit revision info in relation to support and reported issues. Perhaps we should print it during container launch? (available in /home/mig/active-migrid-version.txt)

Both AU and us are pinning the migrid version and the docker-migrid version in our production env anyway.

Just the examples in the docker-migrid repo should be independent.

[benibr]

Can someone explain what the following files are used for please?

• docker-compose_dag.test.yml
• docker-compose_dag.yml

[rasmunk]

Can someone explain what the following files are used for please?

• docker-compose_dag.test.yml
• docker-compose_dag.yml

Yes, these are some legacy configurations that I at some point used for testing the integration with our DAG service. They are not really relevant anymore, since I just use the regular setup now, so feel free to delete them:))

[jonasbardino]

I'd say we'll have the two main examples defaults.env as a development environment and the advanced.env as a production environment. A new value will be added to both of them, so an admin can see those changes and adjust her environment accordingly. Later we can also think about change the major version if a breaking change is introduced.

We need to preserve both a plain and a gdp version to cover SIF-like instances, which are still fundamentally quite different from ERDA-like instances.

[jonasbardino]

Added OS and migrid version info during migrid container launch to ease debug and support.