[Bug]: Upstream compromised? Or is the compromise?

[alerque]

I understand why the author(s) of the analysis of the backdoor being distributed by this project decided not to notify upstream first since it looks like either the upstream is the compromise or at least are compromised themselves, so reporting here first would have done nothing except give the culprit time to wiggle. But the cat is out of the bag now and I see comments all over the place pinging the author. I think it's high time a bug report here notifies people following this tracker there is an issue. Also this serves as a request for a postmortem. Either this project has members that are bad actors or the members have themselves been deeply compromised. Most evidence seems to support the former. If there is evidence of the latter I suggest getting it out there post-haste.

[nickodell]

Some more resources about this:

Red Hat has published a security advisory about this: https://nvd.nist.gov/vuln/detail/CVE-2024-3094

Xe Iaso has an excellent blog post about this: https://xeiaso.net/notes/2024/xz-vuln/

Debian has released a security advisory about this: https://lists.debian.org/debian-security-announce/2024/msg00057.html Users of testing, unstable and experimental are advised to update. Users of stable distributions are unaffected.

Ubuntu has not made an announcement, but I believe they are unaffected: the newest liblzma they ship is 5.4.1.

[FabioPedretti]

Ubuntu published 5.6 in proposed for some time, then it looks like it was reverted: https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2055422 Now they are syncing 5.6.1+really5.4.5-1 to make sure you get safer 5.4, even if you installed 5.6 the days it was in proposed: https://launchpad.net/ubuntu/+source/xz-utils

[alerque]

This comment on Hacker News suggests very heavily the OP is and was a bad actor for a long time. All things considered it definitely sounds like deep review of lots of things (linked binaries, kernel, containers, etc.) is going to be needed and simply rolling back one release might not be enough.

[rubyFeedback]

I think having some kind of blog or text-response may be useful, because after reading:

https://lwn.net/ml/oss-security/20240329155126.kjjfduxw2yrlxgzm@awork3.anarazel.de/

I am confused whether I am possibly affected. I assume no, because I do not use debian, and compile from the released tarballs (source archive) here from github, but even then it may be useful for the xz project to make a few comments (unless that has already happened, but in this case I don't quite know where to read up on this).

[alerque]

This comment on Hacker News has some loose ends of a paper trail pointing not just to @JiaT75 but also @hansjans162 as likely being involved (or possibly an alias of the same actor?). Lots more rabbit holes to go down to root out the mess...

[nickodell]

@rubyFeedback

I think having some kind of blog or text-response may be useful, because after reading:

https://lwn.net/ml/oss-security/20240329155126.kjjfduxw2yrlxgzm@awork3.anarazel.de/

I am confused whether I am possibly affected. I assume no, because I do not use debian, and compile from the released tarballs (source archive) here from github, but even then it may be useful for the xz project to make a few comments (unless that has already happened, but in this case I don't quite know where to read up on this).

It depends on how exactly you downloaded the file. If you went onto the Releases page and downloaded the tarball, then you have the vulnerability. You can see this by opening one of the tar files, and opening m4/build-to-host, and looking at line 63, which is the same line that the openwall announcement says is part of the backdoor.

On the other hand, if you went to the home page, and you clicked on a tag, then clicked "download zip," then you got the files that are in Git, which are different than the release tarball. You would not be affected by the vulnerability in that case. (Unless there is some other backdoor that we haven't found yet.)

[alerque]

Following up on @nickodell, some of the exploit code has been checked into Git and around for much longer than the latest releases, but just stashed in an obfuscated and compressed file that (according to current preliminary analysis) wasn't getting activated until recently. The code to trigger the exploit code was only added in the release tarball and gated behind various conditions.

That doesn't mean other malicious things didn't happen in the past and deeper analysis is still being worked on to see if other triggers might have been present. The presently known and understood trigger was only recently activated and added to the tarballs.

[scy]

you all could start by removing all binary files from this repo, binary files don't belong on git.

I'd say, if you're a compression library, it makes a lot of sense to have binary files containing compressed data in the repo, as part of your test suite. Which is what happened here.

[AdnaneKhan]

Following up on @nickodell, some of the exploit code has been checked into Git and around for much longer than the latest releases, but just stashed in an obfuscated and compressed file that (according to current preliminary analysis) wasn't getting activated until recently. The code to trigger the exploit code was only added in the release tarball and gated behind various conditions.

That doesn't mean other malicious things didn't happen in the past and deeper analysis is still being worked on to see if other triggers might have been present. The presently known and understood trigger was only recently activated and added to the tarballs.

They even added that file to the .gitignore (https://github.com/tukaani-project/xz/commit/4323bc3e0c1e1d2037d5e670a3bf6633e8a3031e). I wonder if the compressed files also placed that file so if you ran tests and built from source you would still get the exploit.

@nickodell maybe there is another backdoor.

[sfalken]

openSUSE has also made their announcement.

https://news.opensuse.org/2024/03/29/xz-backdoor/

[Alcaro]

you all could start by removing all binary files from this repo, binary files don't belong on git.

I'd say, if you're a compression library, it makes a lot of sense to have binary files containing compressed data in the repo, as part of your test suite. Which is what happened here.

Binary files containing compressed data make sense.

35KB binary files containing compressed data, that decompress to apparent nonsense and have no documentation on how they were created, do not make sense.

[P-EB]

I'd be happy to understand why @Larhzu didn't react yet.

[vielmetti]

CVE-2024-3094 is the CVE for this (and that should link to the Github internal page for this, let's see)

[lietu]

I'd be happy to understand why @Larhzu didn't react yet.

Well, it is a national holiday in Finland, and late on a Friday. Some chance they're just out drinking.

[scy]

Binary files containing compressed data make sense.

35KB binary files containing compressed data, that decompress to apparent nonsense and have no documentation on how they were created, do not make sense.

I agree. I was mainly responding to @capicy's overly general statement of "binary files don't belong on git".

[femboywiki]

Does anyone have any information concerning the threat actor such as his identity or country of origin?

[lietu]

Does anyone have any information concerning the threat actor such as his identity or country of origin?

From the sound of it, seems like a fake identity. Unlikely even GitHub can easily find out.

[imadnyc]

@ddvarpdd This being a severe backdoor doesn't excuse racism.

[Alcaro]

Arch Linux has also made an announcement.

https://security.archlinux.org/ASA-202403-1

The problem has been fixed upstream in version 5.6.1.

Considering upstream version 5.6.1 was released three weeks ago, I must say I find that statement rather strange.

e: Fixed at https://gitlab.archlinux.org/archlinux/packaging/packages/xz/-/commit/881385757abdc39d3cfea1c3e34ec09f637424ad , by using the Git release rather than Github. Good job with the embargo, that is a thorougly unsuspicious commit that nevertheless does fix the issue.

[Asday]

@ddvarpdd it's worth looking into what a "false flag" is.

[zb3]

I suspect @JiaT75 just wanted to stay alive.. I hope this is the case.

[imide]

https://archlinux.org/news/the-xz-package-has-been-backdoored/

they made a main website announcement. seems like arch cant be backdoored? i am a moron however

[mathstuf]

The problem has been fixed upstream in version 5.6.1.

The MITRE CVE page doesn't say that; not sure where Arch pulled "5.6.1 fixes it" from malicious code existing "starting with version 5.6.0" in the CVE report.

[ImperatorStorm]

The problem has been fixed upstream in version 5.6.1.

The MITRE CVE page doesn't say that; not sure where Arch pulled "5.6.1 fixes it" from malicious code existing "starting with version 5.6.0" in the CVE report.

Probably just autogenned from https://security.archlinux.org/AVG-2851

[mackal]

The problem has been fixed upstream in version 5.6.1.

The MITRE CVE page doesn't say that; not sure where Arch pulled "5.6.1 fixes it" from malicious code existing "starting with version 5.6.0" in the CVE report.

They're saying their arch specific version 5.6.1-2 fixes it. That switches to pulling from the git tag, which doesn't contain the changes that are only in the release tar ball, so the backdoor can't be activated

[emaste]

FreeBSD is not affected - https://lists.freebsd.org/archives/freebsd-security/2024-March/000248.html

[imide]

I suspect @JiaT75 just wanted to stay alive.. I hope this is the case.

? wym? sorry lmao idk the lore aaah

[ShalokShalom]

As the original email announcement has stated, only deb and rpm distros are targeted. Then, only those who use glibc and link sshd to libsystemd, and libsystemd to liblzma for sd_notify.

Alpine, Arch, NixOS, Void, and so on should all be fine.

I'd be happy to understand why @Larhzu didn't react yet.

Well, it is a national holiday in Finland, and late on a Friday. Some chance they're just out drinking.

Imagine the face that they make, once they wake up tomorrow to this issue. Sober in a second.

[lietu]

So I found it a bit strange that people only mention ssh as potentially affected by liblzma, so I made a quick bash script that loops through files in /usr/lib + /lib checking with ldd for references to liblzma.so.5, and then adds that file to a growing list of potential worries, then scans /usr/bin + /bin + /usr/local/bin for binaries that ldd says uses any of those potentially problematic libraries and .. well .. the list isn't short.

It includes stuff like lxc, NetworkManager, nmap, llvm, qemu, clang, plasma and kde, pgrep, kmod, cupsd, lspci, file, dosbox, smbd, dockerd, xmlsec, overall a lot of generic desktop software as well as incredibly commonplace system utilities. I didn't investigate the vulnerability in detail but with a quick look it seems the potential ramifications of this malicious actor's commit history are a bit larger than initially suspected, especially if ssh was affected just because of some connection to libsystemd, as quite a large % of my system's binaries depend on liblzma.so.5.

https://gist.github.com/lietu/c9b3fc27642d59edb375edc3b4a16c72

EDIT: Writing even slightly more complex BASH is always a minefield, migrated to Python instead to get proper recursive lookups of libraries, and well..

IF there are other malicious pieces included the scope is indeed rather meaningful .. quick maths tells me about 29% of my system binaries, and 50% of the libraries.

https://gist.github.com/lietu/9f01eb7cb47a893341883c03b24682f6

[cpaplham]

@lietu The backdoor that was found only actually loads if argv[0] is /usr/sbin/sshd.

[lietu]

That's at least a small relief, hopefully the full commit history of the account in question gets very rapid and very careful review to find any other potential targets as at least the potential for this was .. a bit more widespread than just sshd

[mackal]

@lietu The backdoor that was found only actually loads if argv[0] is /usr/sbin/sshd.

I don't think it has actually been analyzed enough to verify it was only for sshd, just that it's confirmed for sshd.

[Noah-Leick]

So I found it a bit strange that people only mention ssh as potentially affected by liblzma, so I made a quick bash script that loops through files in /usr/lib + /lib checking with ldd for references to liblzma.so.5, and then adds that file to a growing list of potential worries, then scans /usr/bin + /bin + /usr/local/bin for binaries that ldd says uses any of those potentially problematic libraries and .. well .. the list isn't short.

It includes stuff like lxc, NetworkManager, nmap, llvm, qemu, clang, plasma and kde, pgrep, kmod, cupsd, lspci, file, dosbox, smbd, dockerd, xmlsec, overall a lot of generic desktop software as well as incredibly commonplace system utilities. I didn't investigate the vulnerability in detail but with a quick look it seems the potential ramifications of this malicious actor's commit history are a bit larger than initially suspected, especially if ssh was affected just because of some connection to libsystemd, as quite a large % of my system's binaries depend on liblzma.so.5.

https://gist.github.com/lietu/c9b3fc27642d59edb375edc3b4a16c72

Lmao, a good chunk of my system is going to need to be rebuilt. Gentoo downgraded app-arch/xz-utils to 5.4.6-r1.

Edit: Did not know that xz-utils is not statically linked

[AndrewAmmerlaan]

Lmao, a good chunk of my system is going to need to be rebuilt. Gentoo downgraded app-arch/xz-utils to 5.4.6-r1.

There should be no need to rebuild everything that links with app-arch/xz-utils, the library is not statically linked.

[joeyh]

So I found it a bit strange that people only mention ssh as potentially affected by liblzma

Also the entire build process of several distributions which uses xz files, and also some package formats that use xz files. Any of these could allow a compromised xz to run and do damage.

[cyclone-github]

Script to detect CVE-2024-3094 https://github.com/cyclone-github/scripts/blob/main/xz_cve-2024-3094-detect.sh (This is a fixed and features added version of: https://www.openwall.com/lists/oss-security/2024/03/29/4)

[acook]

As mentioned on https://github.com/google/oss-fuzz/issues/11760 Larhzu and others accounts may have been compromised by collaborators.

Edited to clarify that I am referring to the accounts and not the identities that may or may not be behind them.

[grepwood]

Also consider reviewing Lasse's contributions to the Linux kernel, cause you never know. https://lore.kernel.org/lkml/20240320183846.19475-2-lasse.collin@tukaani.org/

[Jan200101]

As mentioned on google/oss-fuzz#11760 all signs point to Larhzu and others being collaborators.

please do not spread such claims without evidence see https://github.com/google/oss-fuzz/issues/11760#issuecomment-2027738588

[charims]

This issue should probably be locked until such time as there can be a proper response from project leadership. There's a lot of shade and FUD here, and blaming project contributors without evidence isn't helping. I think the urgent situation is handled. Perhaps we can take a step back and wait until we can get a proper response, as I imagine leadership will be quite overwhelmed when they discover all this, and if it were me, it would take me a while to draft a proper well-thought-out response as well as plan my next steps to protect the project.

[luke-beep]

Interesting issue.

[broukema]

There is a live FAQ on this issue:

https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27

And to quote from the current version of the People part:

People

We do not want to speculate on the people behind this project in this document.
This is not a productive use of our time, and law enforcement will be able to handle identifying those responsible.
They are likely patching their systems too.

xz-utils has two maintainers:

    Lasse Collin (Larzhu) who has maintained xz since the beginning (~2009), and before that, lzma-utils.
    Jia Tan (JiaT75) who started contributing to xz in the last 2-2.5 years and gained commit access,
    and then release manager rights, about 1.5 years ago.

Lasse regularly has internet breaks and is on one at the moment, started before this all kicked off.
We believe CISA may be trying to get in contact with him.

[harikattar]

why do we need 700 different compression libraries linked to everything? There's nothing xz does that zlib doesn't for the usecase here. Saving a handful of bytes on already-tiny files isn't worth requiring everything to use all of: libbz2, liblz4, liblzma, libxz and libzstd.

E: that's a whole lot of anonymous cowards hiding behind shitty reaction emojis because they don't understand the issue.

[DanielRuf]

@alerque @broukema thanks for the links, very informative.

@cyclone-github that is very helpful, thanks.

@Asday I agree, most don't know that term and that false flag operations are not that uncommon in the security world.

[mrbid]

why do we need 700 different compression libraries linked to everything? There's nothing xz does that zlib doesn't for the usecase here. Saving a handful of bytes on already-tiny files isn't worth requiring everything to use all of: libbz2, liblz4, liblzma, libxz and libzstd.

E: that's a whole lot of anonymous cowards hiding behind shitty reaction emojis because they don't understand the issue.

You can see who reacted btw, it's not anonymous.

Also zlib as much as I love it... Is dated, zstd is now recommended over zlib for performance reasons, and xz is probably much the same.

[mrbid]

You can see who reacted btw, it's not anonymous.

It's petty

Do I need to remind you that your first post in here was a meme that slights Chinese people.

[carlosrodfern]

Also consider reviewing Lasse's contributions to the Linux kernel, cause you never know. https://lore.kernel.org/lkml/20240320183846.19475-2-lasse.collin@tukaani.org/

Lasse doesn't appear to be involved: https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html

Regarding the kernel, for what I can tell, the JT user did only one code contribution to xz-embedded: https://github.com/tukaani-project/xz-embedded/commit/48355f10e09c5075f3a56031fcbca18b2a708260 That contribution doesn't appear to be compromising anything.

Also, if you do a diffoscope the tarball from github, and the tarball distributed by JT user here: https://xz.tukaani.org/xz-embedded/ you don't see anything relevant included in there.

I think the kernel is clean.

[ShePastAway0]

So should Mac/brew users worry or not? It was definitely shipped with homebrew (5.6.1) but as I understand since there is no systemd which was targeted as well as it is unclear if binaries was targeting x86_x64 only. Could please someone elaborate

[AffSeda]

The scope is probably quite limited, but as Homebrew has rolled the package back just in case; I would update Brew. Although systemd isn't there, glibc does exist on Brew and might be present.

[ShePastAway0]

The scope is probably quite limited, but as Homebrew has rolled the package back just in case; I would update Brew.

could you please elaborate on the scope? and what is the point of updating brew itself instead of removing/downgrading the package? the main question if it was possible to affect Mac Silicons via this "bug", as far as i understand it was some kind of shellcode triggering hence it must be only x86_x64 targeted, correct me if im wrong please