adding package using microdnf no longer works since keycloak version 21

[tobhv]

Before reporting an issue

• [X] I have searched existing issues
• [X] I have reproduced the issue with the latest release

Area

ci

Describe the bug

Before keycloak 21.0.0 I could add packages such as jq using the microdnf command:

microdnf install jq
microdnf: command not found

Next to that in keycloak version before 21.0.0 curl was avilable in the container.

I use curl and jq to define a health check on the docker level and make the check part of containers I build.

health check code put in /usr/local/bin/health.sh:

#!/bin/bash
set -euo pipefail
CHECK=`curl --fail -s http://localhost:8080/health | jq 'select((.status == "UP") or (.checks[].status == "UP"))'`
if [[ -z ${CHECK} ]]; then exit 1;fi

Dockerfile:

ARG QUAY_REGISTRY
ARG KC_VER
FROM ${QUAY_REGISTRY}/keycloak/keycloak:${KC_VER} as builder
ENV KC_METRICS_ENABLED=true
ENV KC_DB=mariadb
ENV KC_VAULT=file
ENV KC_HEALTH_ENABLED=true
RUN /opt/keycloak/bin/kc.sh build
FROM ${QUAY_REGISTRY}/keycloak/keycloak:${KC_VER}
COPY --from=builder /opt/keycloak/lib/quarkus/ /opt/keycloak/lib/quarkus/
COPY rootfs /
USER root
RUN microdnf update -y && microdnf install jq && microdnf clean all; \
    chmod +x /usr/local/bin/health.sh
USER keycloak
HEALTHCHECK CMD /usr/local/bin/health.sh
WORKDIR /opt/keycloak
ENV KC_METRICS_ENABLED=true
ENV KC_HEALTH_ENABLED=true
ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "start", "--optimized"]

Version

21.0.0

Expected behavior

I expect there is an option to install additional packages while the keycloak container is built (or other methods to define a docker health check for keycloak)

Actual behavior

.

How to Reproduce?

.

Anything else?

Info on how I thought to write a health check comes from here: https://www.keycloak.org/server/health and ubi9 also has curl, microdnf onboard: https://hub.docker.com/r/redhat/ubi9-minimal/tags

[sschu]

This is due to https://github.com/keycloak/keycloak/pull/16879 and an intended change to minimise image size and attack surface. You can still install your own dependencies by using a multi-layer build i.e. create your own layer with microdnf, install your dependencies, switch to the Keycloak image and copy your installed dependencies from your own layer.

[tobhv]

thx for the fast reply. will look into it.

[3XC1T3D]

we are issung the same problem. it is not possible to offer two images? one based on ubi9-minimal and the other based on ubi9-micro?

So that the community dont have to rebuild each Dockerfile or CI/CD step.

[sschu]

But aren't you creating your own dockerfile anyways?

[3XC1T3D]

sure we have our own Dockerfile :)

What would your version of following Dockerfile look like with the micro Image? we are unsure how to cover this requirements.

FROM quay.io/keycloak/keycloak:21.0.0

USER root

RUN  microdnf upgrade \
  && microdnf install -y gettext curl \
  && microdnf clean all

We have the problem that we dont know exaclty which packages/files we have to copy from one layer to the other.

[sschu]

You can easily find this out by checking the changes in the layers using for example: https://github.com/wagoodman/dive

[tobhv]

I moved to using ubi9-minimal as a base using the below dockerfile:

ARG QUAY_REGISTRY
ARG REGISTRY
ARG KC_VER
FROM ${QUAY_REGISTRY}/keycloak/keycloak:${KC_VER} as builder
# support meterics and health check
ENV KC_METRICS_ENABLED=true
ENV KC_HEALTH_ENABLED=true
# configure mariadb as the db to be supported
ENV KC_DB=mariadb
# support secrets
ENV KC_VAULT=file
RUN /opt/keycloak/bin/kc.sh build

#use redhat image that supports installing packages (i use curl, jq for the healthcheck)
FROM ${REGISTRY}/redhat/ubi9-minimal

COPY --from=builder /opt/keycloak/ /opt/keycloak/
COPY rootfs /
USER root
RUN microdnf install -y --nodocs --setopt=install_weak_deps=0 \
    java-17-openjdk-headless \
    glibc-langpack-en \
    jq \
    shadow-utils;\
    chmod +x /usr/local/bin/health.sh; \
    useradd -s /sbin/nologin keycloak; \
    microdnf remove libsemanage shadow-utils -y; \
    microdnf clean all;\
    rm -rf \
    /var/lib/dnf
USER keycloak
HEALTHCHECK CMD /usr/local/bin/health.sh
WORKDIR /opt/keycloak
ENV KC_METRICS_ENABLED=true
ENV KC_HEALTH_ENABLED=true
ENV LANG en_US.UTF-8
ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "start", "--optimized"]

Initial testing looks fine with this for new installations, but not for upgrades in my case but that is due to https://github.com/keycloak/keycloak/issues/17285 Reason to use the ubi9-minimal from now on is that I want the option to install packages @runtime in case of troubleshooting.

[3XC1T3D]

Hi @tobhv ,

thank you for your Dockfile example!

I will take a look tomorrow if we can change our CI/CD Setup same lake yours.

But in my opinion the change to the micro image is not worth, with the complicated structure of the Dockerfile afterwards 😅

Best regards

[jangaraj]

My workaround for missing curl is to add static curl:

FROM quay.io/keycloak/keycloak:latest

# see https://github.com/aerogear/keycloak-metrics-spi/releases
ARG KEYCLOAK_METRICS_SPI_RELEASE=2.5.3

# Keycloak 21+ doesn't have curl, workaround for https://github.com/keycloak/keycloak/issues/17273
ADD --chown=root:root https://github.com/moparisthebest/static-curl/releases/latest/download/curl-amd64 /usr/bin/curl
USER root
RUN chmod +x /usr/bin/curl

USER keycloak
RUN \
  curl -L -k https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar \
  -o /tmp/opentelemetry-javaagent.jar && \
  curl -L -k https://github.com/aerogear/keycloak-metrics-spi/releases/download/${KEYCLOAK_METRICS_SPI_RELEASE}/keycloak-metrics-spi-${KEYCLOAK_METRICS_SPI_RELEASE}.jar \
  -o /opt/keycloak/providers/keycloak-metrics-spi.jar

I would recommend this workaround only if you understand security risks of this approach.

I don't like an idea of full image build - it will introduces Docker image management problems. Also this is not a for prod deployment, so I'm happy to accept security risks of this workaround.

[3XC1T3D]

Hi @tobhv

is your Dockerfile working? :sweat_smile:

because i am getting an error on following line:

COPY rootfs /

ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref moby::efr4sdtkotk91o8oq9wjwj76a: "/rootfs": not found

Is this line even valid? :sweat_smile:

EDIT: I think i got it - seems that rootfs it a local dir in your build context :)

[3XC1T3D]

I think our workarround would look like this:

FROM quay.io/keycloak/keycloak:21.0.0 as builder

FROM registry.access.redhat.com/ubi9-minimal

ENV LANG en_US.UTF-8

COPY --from=builder /opt/keycloak/ /opt/keycloak/
USER root

RUN  microdnf upgrade \
  && microdnf install -y gettext \
  && microdnf clean all \
  && rm -rf /var/lib/dnf

WORKDIR /opt/keycloak
# Container will run with an arbitrary user, the following uid is a random default to prevent executing as root
USER 10001

ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]

I hope we are not missing some ENV's from the original keycloak image. Because due to the second FROM Line you "overwrite" all ENV's from the original keycloak image.

[tobhv]

EDIT: I think i got it - seems that rootfs it a local dir in your build context :)

Correct! in this case it just contains my health check script in rootfs/usr/local/bin Regarding your suggested Dockerfile: curious on whether that starts since (in theory) there is no jdk in your resulting image ?

[3XC1T3D]

Hi @tobhv ,

your are right :see_no_evil:

Thank you for the hint!

[tobhv]

no probs.

I hope we are not missing some ENV's from the original keycloak image. Because due to the second FROM Line you "overwrite" all ENV's from the original keycloak image.

My intention is to define environment variables there that a) are different from keycloak's default. b) do not need changing at runtime in my case. Any environment variable can still be defined at runtime (for example db user) using docker-compose / k8s etc.

[vmuzikar]

@tobhv Thanks for reporting this. We need to review our containers guide to reflect changes introduced in https://github.com/keycloak/keycloak/pull/16879. We could also add some example around microdnf.

CC @ASzc

[vmuzikar]

Related: https://github.com/keycloak/keycloak/pull/17292

[DAHAG-ArisNourbakhsh]

16879 is IMO a mistake. This forces users that want to extend the the base image to do super brittle reverse engineered custom images just because they want to augment the system in any small way.

The workaround by @tobhv looks brittle for any changes keycloak will do in the future and already has issues

Initial testing looks fine with this for new installations, but not for upgrades

In the workaround by @3XC1T3D they are already worried that they may have or may not have bulldozed ENV variables and managed to forget the jdk

I can understand why one would lock down the image due to security concerns but I consider this to be overall worse than before.

As context what we do in our dockerfile

• copy another CA into the image and do a update-ca-trust which fails because the underlying utility p11-kit is now gone (probably will have to use a truststore instead from now on)
• and a microdnf install iproute to do some docker swarm required introspection.

FROM quay.io/keycloak/keycloak:21.0.0 as builder

ENV KC_METRICS_ENABLED=true
ENV KC_HEALTH_ENABLED=true
ENV KC_DB=postgres
ENV KC_CACHE_CONFIG_FILE=cache-ispn-jdbc-ping.xml
COPY deployments/. /opt/keycloak/providers/
COPY conf/. /opt/keycloak/conf/
RUN /opt/keycloak/bin/kc.sh build

FROM quay.io/keycloak/keycloak:21.0.0
COPY domentry-net-ca.crt /etc/pki/ca-trust/source/anchors/domentry-net-ca.crt
USER root
RUN chmod 644 /etc/pki/ca-trust/source/anchors/domentry-net-ca.crt \
&& update-ca-trust 
RUN microdnf update -y && microdnf install iproute
COPY custom-entry.sh /opt/keycloak/bin/custom-entry.sh
RUN chmod 777 /opt/keycloak/bin/custom-entry.sh
RUN chown keycloak /opt/keycloak/bin/custom-entry.sh
USER 1000

COPY themes/. /opt/keycloak/themes/
COPY --from=builder /opt/keycloak/lib/quarkus/ /opt/keycloak/lib/quarkus/
WORKDIR /opt/keycloak
COPY deployments/. ./providers/ 
COPY conf/. ./conf/
HEALTHCHECK --start-period=120s CMD curl --fail http://localhost:8080/health || exit 1

ENTRYPOINT ["/opt/keycloak/bin/custom-entry.sh", "start", "--optimized"]

[ASzc]

I'll be taking on the work to update the docs. The example Dockerfiles I've seen posted in comments here are not correct. A couple notes to save everyone effort, until the docs can be updated:

1. There's no need to install curl, just use a Dockerfile ADD instruction, it supports a URL for its source argument.
2. There is no need to revert to ubi9-minimal. There is no need to reinstall all the RPMs in your customized container. Follow the same pattern the keycloak Dockerfile now uses, which is generally:

FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
RUN dnf install -y <insert desired rpms here>
<insert messy rpm-heavy operations here>

FROM quay.io/keycloak/keycloak
COPY --from=ubi-micro-build /some/customization/dir/here /opt/keycloak/

Thanks for your patience

[3XC1T3D]

Hi @ASzc ,

thanks that you take a look at the docs and update it.

1.) i think no one wants a static curl binary link in his Dockerfile. The major people wants the package from the based repository to acchive the latest security fixes etc.

2.) in this example you have to know which folders you have to copy from one layer to the other. In the example from me and @tobhv you dont have to worry about the folders to copy.

that's my opinion :)

[ECX-mpoellinger]

I think removing essential stuff like a package manager is not the best solution when your base idea is to build an individual "own" image.

In the past I just copied our ActiveDirectory root CA into the anchor directory and update-ca-trust. update-ca-trust updates also the java default truststore.

update-ca trust is missing p11-kit -> The option to install it is also gone.

IMHO: This makes everything overcomplicated.

Now we can fight around with keytool instead of the build in tools of RH. And i think also copiing stuff manually for whatever needs to be installed like: executables, libraries etc. will make it even worse

[ASzc]

Hi @3XC1T3D

True, if you're dealing with RPMs, you can't easily COPY all the relevant directories. In my comment just now, I was avoiding the topic of extra RPMs, because that gets really messy. You can easily end up reinstalling all the CVE-bearing RPMs I carefully removed in my PR

ubi-micro defines this chroot pattern for extra rpm installs, that avoids needing to know all the relevant directories:

FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs <rpms go here> --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all

FROM quay.io/keycloak/keycloak
COPY --from=ubi-micro-build /mnt/rootfs /

See ubi-null.sh for my solution, based on this pattern, that strips out the unneeded RPMs. However, I don't think I want to have that script as a documented solution, it was written for keycloak's internal use.

[ASzc]

@ECX-mpoellinger your use case is covered under the example I showed in my first comment. ADD the certificate and RUN update-ca-trust in the first stage, then in the second stage COPY /etc/pki/ (or other more specific directory) across from the first stage

[DAHAG-ArisNourbakhsh]

I docker diffed what a dnf install iproute for example changes in the container. 264 things get touched/created/modified.

(https://gist.github.com/DAHAG-ArisNourbakhsh/9ee4c802ed1f328f5649b972dd4c8295)

~I then have to guess which of these are relevant and copy those over praying I didn't miss something essential that might get invoked in a way I don't know internally in some logic path. This is the least worst solution so far and I probably can live with it but this is the most jank base image experience I've had so far with docker.~

FROM redhat/ubi9 AS ubi-micro-build
RUN dnf install -y iproute

FROM quay.io/keycloak/keycloak:21.0.0

COPY --from=ubi-micro-build /etc/iproute2 /etc/iproute2            
COPY --from=ubi-micro-build /usr/sbin/ip /usr/sbin/ip  
COPY --from=ubi-micro-build /usr/lib64 /usr/lib64          

Edit: I take it back. @ASzc thanks for clarifying

FROM redhat/ubi9 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs iproute --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all

FROM quay.io/keycloak/keycloak:21.0.0
COPY --from=ubi-micro-build /mnt/rootfs /          

[ASzc]

@DAHAG-ArisNourbakhsh Consider the --installroot example from my second comment

[vmuzikar]

@3XC1T3D @DAHAG-ArisNourbakhsh @ECX-mpoellinger Thanks for raising your concerns. Could you please share a bit more details on the motivation for the level of Keycloak image customization? E.g. why specifically do you need iproute tools in a Keycloak image.

Please bear in mind that Keycloak is a security product, hence we're pushing for a secure image with minimum dependencies to make the attack surface as small as possible.

[ASzc]

I would strongly recommend people look very closely at what they've been doing, and find an approach without extra RPMs if possible!

iproute for example, if you're just using it to look at IP addresses, could be easily replaced with:

bash-5.1$ cat /sys/class/net/tap0/address 
ee:ec:da:e1:18:22

[3XC1T3D]

In our case we want to do "only" two tasks:

• install gettext (microdnf install gettext)
• update/upgrade all packages (microdnf upgrade)

we are running security checks over the images with trivy and want to make sure all packages inside the image are up to date.

But anyways. We will take a look on the updated docs and will change our Dockerfiles.

[ASzc]

@3XC1T3D

The good news is that the RPM minimization effort has left only ~40 RPMs, and has drastically reduced the amount of CVEs flagged by Trivy. Before/After:

Total: 72 (UNKNOWN: 0, LOW: 42, MEDIUM: 30, HIGH: 0, CRITICAL: 0)
Total: 13 (UNKNOWN: 0, LOW: 10, MEDIUM: 3, HIGH: 0, CRITICAL: 0)

The bad news is I'm not sure if there's an easy way to do a dnf update now. Obviously the Keycloak image build itself does this, but it's far easier to do then.

I came up with this, but it's a bit clunky:

FROM quay.io/keycloak/keycloak AS original

FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
COPY --from=original / /mnt/rootfs
RUN dnf update --installroot /mnt/rootfs; dnf --installroot /mnt/rootfs clean all

FROM quay.io/keycloak/keycloak
COPY --from=ubi-micro-build /mnt/rootfs /

It may also try to reinstall some of the RPMs I've manually removed outside of dnf, which will open the attack surface back up. It didn't do this when I tried it, but it also found no updates to apply, so that could be why

[3XC1T3D]

Hi @3XC1T3D

True, if you're dealing with RPMs, you can't easily COPY all the relevant directories. In my comment just now, I was avoiding the topic of extra RPMs, because that gets really messy. You can easily end up reinstalling all the CVE-bearing RPMs I carefully removed in my PR

ubi-micro defines this chroot pattern for extra rpm installs, that avoids needing to know all the relevant directories:

FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs <rpms go here> --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all

FROM quay.io/keycloak/keycloak
COPY --from=ubi-micro-build /mnt/rootfs /

See ubi-null.sh for my solution, based on this pattern, that strips out the unneeded RPMs. However, I don't think I want to have that script as a documented solution, it was written for keycloak's internal use.

Hi @ASzc ,

i think we will take your above example to install curl and gettext.

Also i think in future we will skip the dnf update step inside the keycloak image. Your last example is a little bit too clunky to only achieve a 'dnf update'.

Thank you!

[xoxys]

Any recommendation on how to use the docker health check now without curl?

[ASzc]

In the documentation we decided to recommend installing curl, if you have to use HEALTHCHECK and not something else. This adds a lot of CVE surface, but the alternatives aren't great either for other reasons

[xoxys]

This adds a lot of CVE surface, but the alternatives aren't great either for other reasons

What's a real world attack surface added by curl? I mean, even /dev/tcp still exists and can be used for reserve shells and file transfers while for users it's now pretty hard to get native healthcheck support working again for docker or podman because there is no external healthcheck ecosystem evailable.

[ASzc]

The real world access an attacker might have to container binaries+libraries is a question mark, certainly not easy. However it's cleanest to not have them there at all, even if they're not useable. We're also somewhat beholden to automated scanning like Trivy, which simply go by the presence of something in a container.

The decision to remove curl is mostly about its many dependencies, including openssl, which often has CVEs. wget isn't any better because it uses GnuTLS. OpenJDK (and therefore Keycloak) uses NSS, and one security library is enough. Unfortunately I don't know of any CLI tool packaged in RHEL that offers HTTPS via NSS. curl can in theory, but not as compiled in RHEL.

It's unfortunate that Docker hasn't implemented external health checks yet. They're available from other container systems like Kubernetes.

[xoxys]

Then it might help if a java way to execute healthckecks would exist e.g. java run-health.jar.

[vmuzikar]

Then it might help if a java way to execute healthckecks would exist e.g. java run-health.jar.

We were actually considering this but realized that would probably hurt the performance – creating a new JVM for each health check request. Hence we see installing curl as a lesser evil.

[sschu]

That would be an opportunity for graalvm native image builds. ;)

[vmuzikar]

Reopening for backport

[jok2]

As an alternative to a dynamically linked curl i tried a statically linked binary of xh ( https://github.com/ducaale/xh )

My Dockerfile:

FROM quay.io/keycloak/keycloak:21.0.0 as builder

ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
ENV KC_FEATURES=scripts
ENV KC_DB=postgres
ENV KC_HTTP_RELATIVE_PATH=/auth

WORKDIR /opt/keycloak

RUN /opt/keycloak/bin/kc.sh build

FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
RUN export XH_BINDIR=/usr/local/bin && curl -sfL https://raw.githubusercontent.com/ducaale/xh/master/install.sh | sh

FROM quay.io/keycloak/keycloak:21.0.0
COPY --from=builder /opt/keycloak/ /opt/keycloak/
COPY --from=ubi-micro-build /usr/local/bin/xh /usr/local/bin/xh
WORKDIR /opt/keycloak
COPY theme/sinno-prod/ /opt/keycloak/themes/sinno-prod/
COPY theme/sinno-test/ /opt/keycloak/themes/sinno-test/

In my docker-compose.yml i now have a healthcheck section:

        healthcheck:
          test: ["CMD", "/usr/local/bin/xh", " --check-status", "http://127.0.0.1:8080/auth/health/ready"]

[bruegth]

Based on https://ceh51.blogspot.com/2016/07/how-to-open-tcpudp-sockets-bash-shell.html a docker-healthcheck.sh:

#!/bin/bash
exec 3<>/dev/tcp/localhost/8080

echo -e "GET /auth/health/ready HTTP/1.1\nhost: localhost:8080\n" >&3

timeout --preserve-status 1 cat <&3 | grep -m 1 status | grep -m 1 UP
ERROR=$?

exec 3<&-
exec 3>&-

exit $ERROR

What's a real world attack surface added by curl? I mean, even /dev/tcp still exists and can be used for reserve shells and file transfers while for users it's now pretty hard to get native healthcheck support working again

@xoxys yes, it was hard ... ;-)

[xoxys]

I also stumbled across this method, but at least for me, it did not work reliably, as only unencrypted connections are supported. Depending on the KC_PROXY configuration, port 8080 may or may not work. Personally, I am considering using an approach similar to this https://github.com/parkr/go-curl

[HerrDerb]

echo -e "GET /auth/health/ready HTTP/1.1\nhost: localhost:8080\n" >&3

@bruegth Wait, can you explain how that works? 🧐

[xoxys]

@bruegth Wait, can you explain how that works? monocle_face

It's using the pseudo-device /dev/tcp (which is a Bash feature) to perform a TCP connection, see https://tldp.org/LDP/abs/html/devref1.html

3<> is opening a read/write file descriptor for the socket communication, see https://tldp.org/LDP/abs/html/io-redirection.html

Finally, grep is used to check if the strings "status" and "UP" appears in the response to exit the script successfully or with a none-zero exit code in case of an error.

[HerrDerb]

@xoxys Thanks. It does work, and is the perfect solution for me (y)

[t-schuster]

To avoid opening a duplicate:

The issue prevents installing custom certificate-authorities reliably into the container and in our build pipeline also stops usage of curl/wget to fetch custom providers in the Dockerfile to provide them for the build.

Allowing p11-kit to be installed would help a lot here instead of having to create a brittle and fragile script to manually put the certificate stores into proper order.

[ASzc]

@t-schuster Your use cases are covered, as detailed in previous comments in this thread.

1. Do your certificate work in one stage FROM ubi9, where all the packages you need are available, then copy across /etc/pki into the second stage.
2. To install custom provider JARs, just use ADD --chown=keycloak:keycloak. You don't need curl to access URLs.

Both of these use cases are now included in the latest container documentation, but we're still working on the backport to the KC 21 docs.

[ghenadiibatalski]

@ASzc sorry for the stupid question. We've used an envsubst (microdnf install gettext) form the customized entrypoint.sh and replaced some details from runtime env vars eg.

if [ -e /realms/realm-template.json ] ; then
  envsubst < /realms/realm-template.json > /tmp/realm.json
fi

and

ENV KEYCLOAK_IMPORT /tmp/realm.json

what is the best way for such manipulations (must not be only the import json, but some other things used by entrypoint)?

Regards, Gena

[zyv]

@3XC1T3D @DAHAG-ArisNourbakhsh @ECX-mpoellinger Thanks for raising your concerns. Could you please share a bit more details on the motivation for the level of Keycloak image customization?

We have been using microdnf to install Python stack, because we had to write a management harness in Python to make it possible to develop our custom realm-dependant themes for Keycloak and manage Keycloak in production. Also, this harness is building different types of our custom Keycloak extension modules and prepares an image where they come pre-installed.

The removal of microdnf has made things massively more complicated for us. Of course, we can repackage Keycloak ourselves instead of using the official image, but this is another level complexity and customisation depth. We will try to find a middle ground based on the posts in this thread :(

[kantum]

Going back to 20, let me know when documentation is up-to-date, just need curl for healthcheck, wasted too much time....

[vmuzikar]

The docs on the website will be updated with the next micro release. In the mean time, please check it in the PR that is attached to this issue.

[ASzc]

@ghenadiibatalski If it truely has to be at runtime, then you can use this to install gettext

Consider working the changes into your multi-stage image if possible. That way it's possible to avoid installing additional RPMs in the actual container image. See the COPY /etc/pki example in the same part of the documentation I linked above.