High severity vulnerability in `libexpat` packages pulled into Alpine

[sxa]

Hi.

I had a discussion on an issue with @yosifkit recently about an openssl issue on Ubuntu.

A user of ours has reported a high severity vulnerability with the eclipse-temurin images, and the dockerhub site for the image is showing it as a high (CVSS7.5)

Back in 2022 there was a rebuild done for us by @tianon to resolve another vulnerability in the same package, but as per yosifkit's comment in the first referenced issue it would be ideal if all images using libexpat which are showing this vulnerability could get updated so they get 2.6.0-r0 unless there is a reason why this should not be considered a sufficiently high severity.

Since I don't think the Alpine images necessarily have a periodic refresh, and Alpine do not seem to want to trigger anything under their control for this and we have no mechanism to do so, despite the following suggestion to the raiser in that iAlpine ssue:

If people are getting security warnings because they use eclipse-temurin, they probably need to wait for them to release a newer image that use the new expat version you mentioned.

I suspect the person making that comment may not be aware that it is not under our control to trigger such a rebuild. Given all of the above, is this a valid justification for a trigger of a rebuild of ours, and possibly other people's, Alpine images?

[tianon]

It's interesting/unique that Alpine's got this one fixed already, as both Debian and Ubuntu don't even have commentary on it yet:

• https://security-tracker.debian.org/tracker/CVE-2023-52425

• https://ubuntu.com/security/CVE-2023-52425

• (vs) https://security.alpinelinux.org/vuln/CVE-2023-52425

[tianon]

libexpat through 2.5.0 allows a denial of service (resource consumption) because many full reparsings are required in the case of a large token for which multiple buffer fills are needed.

Any idea how widespread use/indirect invocation of libexpat inside Eclipse Temurin is?

[sxa]

It's pulled in as a dependency of fontconfig I believe. The amount that is used works be purely dependent on how many users' java applications running on top of us trigger anything relating to that functionality. Even on our Alpine build, which are headless, some of the java APIs still use some such operations that you might initially think we're only for headful JDKs.

I'm not sure that necessarily helps, but hopefully that makes sense.

[sxa]

Ref Ubuntu/Debian while I haven't looked into it myself, the user who reported this to us did assert:

The debian/ubuntu based images got rebuilt a few days ago so the problem is no longer present there

But as you say a little surprising if they haven't noted anything related to that specific CVE on they web pages.

[tianon]

The defect is definitely still present on any Debian-based images -- https://tracker.debian.org/pkg/expat hasn't been uploaded to stable or stable-security (only unstable) since 2022. :sweat_smile:

The situation is the same for Ubuntu Jammy and older on https://launchpad.net/ubuntu/+source/expat (slightly newer on Mantic, but certainly not for this CVE).

[AB-xdev]

Ref Ubuntu/Debian while I haven't looked into it myself, the user who reported this to us did assert:

The debian/ubuntu based images got rebuilt a few days ago so the problem is no longer present there

Disclaimer:

1. The user is me
2. I thought the vulnerability is no longer present there because there it is not listed in vulnerabilities section of the ubuntu variant of temurin, but it could be quite possible that the CVE scanner just doesn't find it.

[sxa]

The defect is definitely still present on any Debian-based images -- https://tracker.debian.org/pkg/expat hasn't been uploaded to stable or stable-security (only unstable) since 2022. 😅

Notwithstanding how utilised the particular function might be by users of the Eclipse Temurin image specifically, at this point my gut feel is that Debian/Ubuntu are possibly a bit behind on this in terms of not assigning a severity rating / providing a patch of some sort and the analysis that is available from Alpine is likely valid which would suggest that it should likely be one of those situations where it is patched through a rebuild for everyone who is pulling it into their images who could be affected by a potential DoS and may not be aware of it. What do you think?

Is this a valid reason to perform a forced rebuild of the Alpine images? I'm not sure what dockerhub's typical policy is here but I guess it comes down to whether "high" severity being flagged on the dockerhub pages is typically enough to warrant such a rebuild for users or whether dockerhub would typically only choose to do that for "Critical" level severities.

[hartwork]

It's interesting/unique that Alpine's got this one fixed already, as both Debian and Ubuntu don't even have commentary on it yet:

• https://security-tracker.debian.org/tracker/CVE-2023-52425

• https://ubuntu.com/security/CVE-2023-52425

• (vs) https://security.alpinelinux.org/vuln/CVE-2023-52425

@tianon to be fair, Debian does have commentary and has Expat at 2.6.0 in sid, and they were very quick about it.

@sxa hello, Expat upstream here. I'm having trouble really understanding the title and the mission of this ticket. Could you help me better understand?

[tianon]

Right, sorry, I meant specifically that the Debian Security Team has not added explicit additional commentary ("does not apply to Debian", "not considered critical", etc; they only have the upstream links in their notes field) -- the maintainer fixing it in unstable was "easy" because it's just a version bump (and presumably had few/no breaking changes) where fixing it in stable/oldstable is harder because it has to be a targeted backport of the security patch (and thus typically comes with some degree of "is the potential of brekaing (old)stable higher than the potential of users hitting the vulnerable code" that's typically done by the Debian Security Team :heart:)

[sxa]

@sxa hello, Expat upstream here. I'm having trouble really understanding the title and the mission of this ticket. Could you help me better understand?

1. We (eclipse-temurin) have official docker images based on Alpine which are currently causing a high severity vulnerability to be flagged.
2. Alpine's base images (since they are as small as possible) don't include libexpat.
3. Our Dockerfiles pull in libexpat (as a depedency of something else) so it pulls in the "latest" from the Alpine repositories at the time the dockerfile is built (under control of dockerhub - we can't trigger a rebuild).
4. Alpine has updated their repositories with the new version, but since our images have not been rebuilt sine the Alpine repositories have been updated we do not have the fix.

@hartwork As an expat maintainer I expect is probably out of your control since you have a fix and it's already in the appropriate repositories. This is more an issue about dockerhub's Dockerfile rebuild policy, or whether Alpine should be able to force a refresh of their base image for something like this :-)

[hartwork]

Hi @sxa I understand that the alpine image does not include expat and was last pushed a month ago and that Expat in Alpine repos is at 2.6.0 by now. What I do not understand why in stage (3) the build of eclipse-temurin would pull in Expat <2.6.0 if Alpine has more recent OR why stage (3) does not update Alpine package indices — apk update? — before installing what pulls in Expat. Wouldn't that solve the problem despite Alpine's image being a month old? Is (4) still status quo? What is keeping eclipse-temurin from a rebuild with the update-package-index step added?

[AB-xdev]

(3) the build of eclipse-temurin would pull in Expat <2.6.0

Because the image was built like a month ago when only version 2.5.0 was available.

(3) does not update Alpine package indices — apk update

AFAIK because this is a bad practice to do in Docker images as it will bloat up your layers (and it also may cause incompatibilities).

[sxa]

What is keeping eclipse-temurin from a rebuild with the update-package-index step added?

The statement at the end of 3 is the reason - such rebuilds are not under eclipse-temurin's control for official images. That's why I've raised this against dockerhub:

the time the dockerfile is built (under control of dockerhub - we can't trigger a rebuild).

[hartwork]

(3) does not update Alpine package indices — apk update

AFAIK because this is a bad practice to do in Docker images as it will bloat up your layers

@AB-xdev sounds like security OR saving a few megabytes. Would be a clear call for security with me. It's definitively what I'd recommend to people and what I do in the images I control and deploy myself, e.g.:

• https://github.com/hartwork/docker-ssl-reverse-proxy/blob/04e7240dd844089eb304f35277f4ac9a6878bb93/Dockerfile#L11
• https://github.com/hartwork/jawanndenn/blob/6ccac62f33bf0f117026ec563bef76530386c9c5/Dockerfile#L37
• https://github.com/hartwork/wnpp.debian.net/blob/62cbb81c4ff45d6825c010a87a6811649dace666/docker/Dockerfile#L44

(and it also may cause incompatibilities).

@AB-xdev are we talking security updates or rolling-release distro updates here? Security-only updates should normally not break things, and bypassing security updates comes at a cost.

What is keeping eclipse-temurin from a rebuild with the update-package-index step added?

The statement at the end of 3 is the reason - such rebuilds are not under eclipse-temurin's control for official images. That's why I've raised this against dockerhub:

the time the dockerfile is built (under control of dockerhub - we can't trigger a rebuild).

@sxa what if you force-changed the related Dockerfile, say by adding && true somewhere — would that trigger a rebuild with dockerhub? What kind of events do they do automatic rebuilds on?

[sxa]

What is keeping eclipse-temurin from a rebuild with the update-package-index step added?

The statement at the end of 3 is the reason - such rebuilds are not under eclipse-temurin's control for official images. That's why I've raised this against dockerhub:

the time the dockerfile is built (under control of dockerhub - we can't trigger a rebuild).

@sxa what if you force-changed the related Dockerfile, say by adding && true somewhere — would that trigger a rebuild?

Almost certainly yes but I want to avoid doing that for every security issue unless strictly necessary and I'm not sure @tianon etc. would really want us to do that either. The main reason for using official images is to avoid us having to maintain and patch the images for such things, and also with the hope that others in the same situation can take advantage of the same process without requiring each project to do what is fundamentally a pointless change to their files (which also requires a PR, approval etc. so it's more manual work on both sides for every project that chooses that path so isn't the best solution from a scalability perspective).

What kind of events do they do automatic rebuilds on?

Obviously I can't say for sure, but I believe for Alpine specifically it's only when the base image is updated, but they can force an update when a critical security flaw is detected as per their FAQ.

I'm looking to agree on a consistent process here for security fixes to get in, not implement a one-off "hack" around it if at all possible, which is why I'm having the discussion in this repository.

[hartwork]

I'm looking to agree on a consistent process here for security fixes to get in, not implement a one-off "hack" around it if at all possible, which is why I'm having the discussion in this repository.

@sxa I see. I think what I would consider doing myself in your situation is setting up a GitHub-Actions based cronjob that produces artificial diff to trigger rebuilds whenever there are not-yet-applied updates found during the build. The thing could provide pull requests for you so a human would acknowledge or reject each case, and once that automation is built you can stop thinking about it altogether and be sure that your security fixes are lagging no more than the cron frequency. It sure won't win prices in purity, but I wouldn't be surprise if this is already the best you can get from a results perspective with regard to being secure, control over lag, and a consistent process. I'll be curious for sure to hear about anything better that you'll find.

PS: Here's an example of how GitHub Actions creates pull requests for me, to update outdated pre-commit hooks in this very case: https://github.com/hartwork/git-delete-merged-branches/blob/master/.github/workflows/pre-commit-detect-outdated.yml . Or the same thing running cargo update: https://github.com/hartwork/rust-for-it/blob/main/.github/workflows/cargo_lock_autoupdate.yml . So that's the first 30% of the task for free in case you do decide to take that route :smiley:

[sxa]

Images have been refreshed by dockerhub, so this can now be closed.