Extend the release pipeline by publishing Docker image

[amotl]

Hi there,

in order to make this package usable for all people on Docker without further ado, we want to build and publish corresponding Docker images.

While we will start with the basics, we might want to look at building extended images containing more nifty dependencies like GDAL which are not easy to setup on Windows.

With kind regards, Andreas.

[amotl]

I've tried to get something together within the test-docker branch. It's just a rough sketch, but it aims to

a) Quickly test the Docker image by invoking a basic command wetterdienst about parameters. b) Upload the image to the GitHub Package Registry.

[amotl]

After doing that with the release 0.7.1-testing (we might want to delete that afterwards), I learned from [1] that images published to the GitHub Package Registry will not be available for public consumption without authenticating. However, it will be possible when using the GitHub Container Registry.

@nightlark said at [2]:

The recently announced GitHub Container Registry seems to be the answer to this issue. It allows anonymous read access, and it looks like it could provide more control for managing packages. Some useful docs pages about it:

• About GHCR
• Migrating to GHCR

The main downside is that it looks like a personal access token needs to be created to push to a GitHub org/user-level container repository, so a GITHUB_TOKEN with GitHub repository scope isn’t enough.

[1] https://github.community/t/docker-pull-from-public-github-package-registry-fail-with-no-basic-auth-credentials-error/16358 [2] https://github.community/t/docker-pull-from-public-github-package-registry-fail-with-no-basic-auth-credentials-error/16358/77

[amotl]

Switching to ghcr.io

With more adjustments, I am now using docker-publish-standard.yml in order to push images to the GitHub Container Registry.

Apparently, it already pushed an image to ghcr.io/earthobservations/wetterdienst-standard, see [1].

Permission adjustments

However, docker pull ghcr.io/earthobservations/wetterdienst-standard fails yet. This is probably because

When you first publish a package, the default visibility is private and only you can see the package. [2]

The documentation at [2] tells us how to make this package public:

I would need your help on this, @gutzbenj.

[1] https://github.com/earthobservations/wetterdienst/runs/1141377767?check_suite_focus=true [2] https://docs.github.com/en/packages/managing-container-images-with-github-container-registry/configuring-access-control-and-visibility-for-container-images#configuring-visibility-of-container-images-for-an-organization

[amotl]

With #172, two images have been published. When the access permissions will be relaxed, this should work without further ado:

docker pull ghcr.io/earthobservations/wetterdienst-standard:test-docker
docker pull ghcr.io/earthobservations/wetterdienst-full:test-docker

[gutzbenj]

Switching to ghcr.io

With more adjustments, I am now using docker-publish-standard.yml in order to push images to the GitHub Container Registry.

Apparently, it already pushed an image to ghcr.io/earthobservations/wetterdienst-standard, see [1].

Permission adjustments

However, docker pull ghcr.io/earthobservations/wetterdienst-standard fails yet. This is probably because

When you first publish a package, the default visibility is private and only you can see the package. [2]

The documentation at [2] tells us how to make this package public:

I would need your help on this, @gutzbenj.

[1] https://github.com/earthobservations/wetterdienst/runs/1141377767?check_suite_focus=true [2] https://docs.github.com/en/packages/managing-container-images-with-github-container-registry/configuring-access-control-and-visibility-for-container-images#configuring-visibility-of-container-images-for-an-organization

Done

[gutzbenj]

The release pipeline seems to work now! Very nice! However we should add a link to the registry as it is not given within the repository itself.

[amotl]

Very nice!

Dito. Thanks for quickly adjusting the access permissions.

We should add a link to the registry.

Sure, we should update the documentation.

The contents of the "Setup > Docker" section should move to "Development > Docker" and should be replaced by an updated version pointing to the image(s) on ghcr.io.

docker run -ti ghcr.io/earthobservations/wetterdienst-standard wetterdienst --version

Edit: Done with #176.

[amotl]

Release procedure

Status quo

I also want to note that the Dockerfile used for building the images will just invoke pip install wetterdienst. Thus, it will always use the latest release available on PyPI.

On the other hand, the original Dockerfile uses the content of the current working tree for building images. This is differently but obviously preferred for development sandbox situations.

Discussion

We might want to discuss this important detail. When we would leave it like it is, we would need to ensure the Docker image building jobs are executed strictly after the job responsible for uploading this package to PyPI. As far as I know, it is not possible to synchronize execution across different GitHub Actions so we should do something about it?

Please let me know what your thoughts are about this. We should really get this straight.

Proposal

The best thing would probably be to build an sdist package from the sources within the current working tree, copy it into the Docker container and install it there. When doing it that way, we would not need to install and use Poetry within the Docker container. I am preferring a more production-like setup here and Poetry is clearly a development tool which should not be shipped with production images.

[amotl]

We would need to ensure the Docker image building jobs are executed after the job responsible for uploading this package to PyPI.

Otherwise, we will still have the old release in the latest Docker images.

Example

After just releasing 0.8.0, we still have 0.7.0 within the Docker images:

docker pull ghcr.io/earthobservations/wetterdienst-standard
alias wetterdienst='docker run -ti ghcr.io/earthobservations/wetterdienst-standard wetterdienst'

$ wetterdienst --version
wetterdienst 0.7.0