Reduce dependencies of libsystemd

[Lastique]

Component

systemd

Is your feature request related to a problem? Please describe

The recent sshd/xz backdoor fiasco (CVE-2024-3094) has shown that the extra dependencies introduced by libsystemd may be the source of vulnerabilities in core and sensitive components on the system. libsystemd is being linked into all systemd services, as well as any third party services that intend to interact with systemd through the C API. For example, any service that wishes to notify systemd of its startup and termination would call sd_notify family of functions. Any process that wishes to write logs to systemd-journal would call sd_journal_print family of functions. And so on.

Describe the solution you'd like

This issue is asking to reduce the dependencies of libsystemd to the bare minimum, which is libc. Currently, on Kubuntu 22.04 (libsystemd0 version 249.11-0ubuntu3.12) ldd libsystemd.so.0 shows:

        linux-vdso.so.1 (0x00007fff97fbd000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f9519a77000)
        libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f95199a8000)
        liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f9519988000)
        libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007f951997d000)
        libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f951983f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9519600000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f9519b93000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f95195da000)

I believe, most of these dependencies are not necessary to implement core libsystemd functions, like those mentioned above.

This issue may mean splitting libsystemd to multiple libraries implementing different APIs, one of which, say, libsystemd-core, would only depend on libc, and other, more specialized libraries, would add other dependencies. Also, if some of the dependencies are only needed by certain systemd services, move the dependencies to those services.

The ultimate effect of this should be reduced attack surface and improved system security.

Describe alternatives you've considered

For some APIs, there are documented underlying protocols (e.g. there is native journal protocol). Third party processes could be implementing those protocols to interact with systemd. However, this approach is counter-productive and error-prone, as every process would need to implement the same protocol, and may do it incorrectly. This would also make future evolution of these protocols more difficult as there would be many implementations of it that would need to be upgraded. Furthermore, this does not solve the problem for systemd itself, as the dependencies remain loaded into every systemd process, whether they are needed or not.

The systemd version you checked that didn't have the feature you are asking for

249

[DaanDeMeyer]

In the next release all the compression libraries and gcrypt will be dlopen() style dependencies which means that the libraries will only be loaded if the relevant functionality in libsystemd is actually used by the service.

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/4VRKIAQAXGXU7C7A2ERO7SN3WTC5ML33/ gives an overview of why splitting libsystemd is not trivial and involves other tradeoffs.

[Lastique]

I think, dlopen-style usage would make the problem more difficult to diagnose, as it isn't obvious which APIs use which third-party libraries. It might be an improvement over the status quo in the immediate future, but I would still prefer the proper solution, where the dependencies are linked in and libsystemd is split. I understand this may mean a lot of refactoring work, but this is what needs to be done, IMO.

[Conan-Kudo]

I do think we need to start strongly considering splitting libsystemd back up into component libraries. Yes, this will be rather painful because there have been no architecture boundaries for the library code for 10 years now, but this incident has shown that the architecture of integration matters a lot.

The dlopen() path doesn't actually help us because it isn't changing the architecture, it just obscures what systemd uses to both users and maintainers.

[ncopa]

I would love to see the big mono-repo split up. There are many good things in systemd that I'd love to use in Alpine Linux but can not, due to only GNU libc is supported. A few things that could be nice to have separated out are:

• systemd-boot (gummiboot)
• sd_notify
• udev

It would be very nice if those could be built independently by init systems that have musl libc support.

[Conan-Kudo]

I don't think that's realistic or desirable by the contributors. And none of us in this issue are asking for that. But having a modular architecture does not necessarily require a polyrepo setup. And Alpine can still take the source code and build only the components they want to use.

As for Musl libc support, I think that's largely dependent on beefing up musl to support the missing primitives. Nobody here is against it as long as the stuff needed by systemd is present on musl-based systems.

[YHNdnzj]

@ncopa That's a different topic, and I don't think it's desirable. Maintaining components in one repo has a major benefit that allows us to reuse basic/ and shared/ util libs. Each component can still be built standalone, one repo or not.

Regarding musl, people from postmarketOS are working on it AFAIK. But that's pretty irrelevant too.

[jbwyatt4]

Some ignorant questions:

Why does libsystemd need 3 different compression libraries?

Can they be reduced?

(As I was typing this the above MR for dynamically loading compression libraries was merged.)

[bluca]

We are not going to split again the library, because that just creates usability, maintainability and integration problems for no gains. The dependencies will all be optional via dlopen, and I'll also document that using sd_journal APIs will result in optional dependencies to be loaded.

[ncopa]

I don't think that's realistic or desirable by the contributors. And none of us in this issue are asking for that. But having a modular architecture does not necessarily require a polyrepo setup.

Fair enough. My thinking here was poly repo setup would likely help enforcing a modular architecture if that was a goal. I just wanted to mention that it would also be beneficial for non-supported platforms.

And Alpine can still take the source code and build only the components they want to use.

Sure, I'm just saying it would be nice to have it more separated.

Nobody here is against it

Being against and supporting something are two different things.

as long as the stuff needed by systemd is present on musl-based systems.

This means "not supported".

Thanks for listening.

[bluca]

This is the dependency tree with a full-feature build and the pending PRs merged:

build/libsystemd.so.0 (interpreter => None)
    libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
    ld-linux-x86-64.so.2 => /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2

There's a TODO item to replace libcap with ioctls, maybe this is a good time to look into that too? Any volunteers?

[bluca]

After lunch I'll also work on adding an MIT-0 example to the manpage on how to reimplement sd_notify in C, so that it is copy/paste ready. The notify protocol is stable and trivial for the manager <-> service case, and if one wants to prioritize avoiding dependencies we should facilitate that.

[YHNdnzj]

The dlopen() path doesn't actually help us because it isn't changing the architecture, it just obscures what systemd uses to both users and maintainers.

Well, nobody is against adding dlopen-ed deps to ELF metadata (https://github.com/systemd/systemd/pull/31131#issuecomment-1917618062). But there's no standard on this and I doubt we'll ever get that soon.

[kampka]

For some APIs, there are documented underlying protocols (e.g. there is native journal protocol). Third party processes could be implementing those protocols to interact with systemd. However, this approach is counter-productive and error-prone, as every process would need to implement the same protocol, and may do it incorrectly. This would also make future evolution of these protocols more difficult as there would be many implementations of it that would need to be upgraded.

I'm sorry if I'm sidelining the discussion a bit, but if I understand @poettering here correctly, these protocols are documented with the explicit intent ("on purpose") to enable third party projects to re-implement them as they see fit. As such, any evolution of these protocols would surely have to take that into account already. In his post he is talking about sd_notify, which is of course a much simpler protocol that the journal native protocol, but I understood the point to apply any Systemd protocol documented in such a way. If my understanding is wrong, if would be nice if you could clarify the Systemd position with regards to these documentations, so I and probably others don't program myself into a long-term corner. Thanks :)

[bluca]

stable interfaces and protocols are already documented at: https://systemd.io/PORTABILITY_AND_STABILITY/

[Lastique]

I'm sorry if I'm sidelining the discussion a bit, but if I understand @poettering here correctly, these protocols are documented with the explicit intent ("on purpose") to enable third party projects to re-implement them as they see fit.

The ability to reimplement the protocols is good and very welcome, but there is benefit in having a reference implementation that is easy to use in third party services without much downsides, like unnecessary dependencies. This relieves those services from having to implement, test and maintain their own implementations, however simple it may be. My understanding is that libsystemd was supposed to be that reference implementation.

[bluca]

After lunch I'll also work on adding an MIT-0 example to the manpage on how to reimplement sd_notify in C, so that it is copy/paste ready. The notify protocol is stable and trivial for the manager <-> service case, and if one wants to prioritize avoiding dependencies we should facilitate that.

--> https://github.com/systemd/systemd/pull/32030

[bhaible]

After lunch I'll also work on adding an MIT-0 example to the manpage on how to reimplement sd_notify in C, so that it is copy/paste ready. The notify protocol is stable and trivial for the manager <-> service case, and if one wants to prioritize avoiding dependencies we should facilitate that.

Even better than example code in a documentation would be a function that gets linked into the program, without actually adding libsystemd.so.N as a runtime dependency. Recall that libc.so is in fact a linker script:

OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /lib/x86_64-linux-gnu/libc.so.6 /usr/lib/x86_64-linux-gnu/libc_nonshared.a  AS_NEEDED ( /lib64/ld-linux-x86-64.so.2 ) )

I imagine that a similar linker script could be used for libsystemd.so:

OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /usr/lib/x86_64-linux-gnu/libsystemd_nonshared.a  AS_NEEDED ( /lib/x86_64-linux-gnu/libsystemd.so.0  ) )

libsystemd_nonshared.a would contain sd_notify and all the other "simple" API functions that don't need other shared library dependencies.

libsystemd.so.0 would contain everything, including these "simple" API functions (for binary backward compatibility).

[bluca]

No, we are not adding new libraries. If you want convenience then just use the existing library, otherwise put in some effort and use the copy pastable example.

[Conan-Kudo]

We are not going to split again the library, because that just creates usability, maintainability and integration problems for no gains.

Clearly there are gains or it wouldn't be brought up all the time, and especially now.

The dependencies will all be optional via dlopen, and I'll also document that using sd_journal APIs will result in optional dependencies to be loaded.

And you are free to do this, even though it doesn't fix any of the problems people are actually talking about.

[YHNdnzj]

We are not going to split again the library, because that just creates usability, maintainability and integration problems for no gains.

Clearly there are gains or it wouldn't be brought up all the time, and especially now.

The dependencies will all be optional via dlopen, and I'll also document that using sd_journal APIs will result in optional dependencies to be loaded.

And you are free to do this, even though it doesn't fix any of the problems people are actually talking about.

Sorry, but I really don't see the point here. If people don't use sd-journal in the first place, no extra deps will be loaded. If sd-journal is used and we split the library, you are still linking to sd-journal anyway. So what's the real gain? Essentially this is status quo, except that by splitting libsystemd we create a huge compat break.

[YHNdnzj]

And only some of sd-journal functions actually triggers dlopen, so in the current impl people get minimized dependencies in memory. If we step away from dlopen again that's not the case anymore.

[YHNdnzj]

So as mentioned - if the request is about adding the list of potentially dlopen-ed deps to ELF metadata, no objections exist and the only obstacle lies in making this an actual standard. If this is about splitting the library you only get compat break and maintainability issues for no real gain.

[Lastique]

Sorry, but I really don't see the point here. If people don't use sd-journal in the first place, no extra deps will be loaded. If sd-journal is used and we split the library, you are still linking to sd-journal anyway. So what's the real gain?

I don't quite understand why user's application calling e.g. sd_journal_print should require any of the compression libraries. Now or in the future. My understanding is that this and similar functions only implement the native journal protocol, which has nothing to do with data compression.

If data compression is involved then please provide a new set of functions that don't require compression. And document this fact so that users can rely on it.

[Lastique]

And only some of sd-journal functions actually triggers dlopen, so in the current impl people get minimized dependencies in memory. If we step away from dlopen again that's not the case anymore.

The way it is now documented in https://github.com/systemd/systemd/pull/32030, any sd_journal functions are allowed to call dlopen, including sd_journal_print. From the user's perspective, this is equivalent to when libsystemd was linked with compression libraries.

[Lastique]

So as mentioned - if the request is about adding the list of potentially dlopen-ed deps to ELF metadata, no objections exist and the only obstacle lies in making this an actual standard. If this is about splitting the library you only get compat break and maintainability issues for no real gain.

The request in this issue is to reduce dependencies of libsystemd. From this perspective, explicit dlopen calls are equivalent to ELF metadata (if I understand this correctly) and pose the same sort of solution, which is to dynamically load libraries as needed. The discussion between dlopen and ELF metadata is irrelevant in this context. Another solution is to split libsystemd into multiple libraries and manage dependencies this way. It does provide benefits that were discussed above, like explicit dependency tracking and avoiding unnecessary dependencies.

[YHNdnzj]

The way it is now documented in #32030, any sd_journal functions are allowed to call dlopen, including sd_journal_print. From the user's perspective, this is equivalent to when libsystemd was linked with compression libraries.

How is this any different from splitting sd-journal into a dedicated library and linking to that?? I don't understand why the discussion is going in circles at all.

[YHNdnzj]

The discussion between dlopen and ELF metadata is irrelevant in this context.

It's requested by several people so that ldd and friends can list all potentially used libraries.

[Lastique]

How is this any different from splitting sd-journal into a dedicated library and linking to that?? I don't understand why the discussion is going in circles at all.

I did not suggest any specific splitting boundaries. But given that the goal is to reduce dependencies, the split could be made in such a way that sd_journal functions that don't require compression libraries could be linked separately from those that do. I mentioned libsystemd-core as an example; in this example all functions that don't require extra dependencies would be in this library, including sd_notify and parts of sd_journal.

[bluca]

As already explained, we are not going to split the library. There is no point in wasting time going over and over again the same points that have already been clarified.

[ericcurtin]

A nice thing about this change is it could help minimize initrd size (as you can exclude the libraries that will never be dlopened), optimising boot times. Some initrd's want systemd as a basic init system and not have all the bells and whistles included early boot.

[RussNelson]

Shouldn't libsystemd be entirely eliminated and replaced by APIs that go through IPC? Remember, if you've found one security lapse, there are ten you haven't found yet.

[ZanderBrown]

You seem to have forgotten to put the /s on that?

[arvidjaar]

it could help minimize initrd size (as you can exclude the libraries that will never be dlopened)

How initrd generator is supposed to know which libraries will be needed?

[poettering]

I do think we need to start strongly considering splitting libsystemd back up into component libraries. Yes, this will be rather painful because there have been no architecture boundaries for the library code for 10 years now, but this incident has shown that the architecture of integration matters a lot.

Sorry, but no. Splitting this up makes a mess, since it makes sharing internal code much harder. You either have to make all our internal helpers public symbols (which means namespacing issues, ABI stability and all that fuck), or you "statically" compile the shared libraries, i.e. you duplicate the internal helpers in each split up library, exploding the size on disk. Which is also terrible.

I am hence vehemently against splitting this up. It comes with major drawbacks.

I'd recommend apps to just embed their own minimal version of the sd_notify() protocol, in their native language. It's trivial, and nowadays even in crazy languages like Java you have AF_UNIX support that is sufficient to implement it in a few lines of code.

I think the dlopen() thing is fine. It makes things just work: the libs are loaded when they are needed, not before. Yes, dlopen() makes the deps less discoverable for tools like ldd. But one shouldn't conflate two different concepts:

1. loading libs only on demand + handle missing deps gracefully
2. making these deps discoverable

The 2nd item is fully valid, but there's no need to insist in DT_NEEDED for that. We could relatively easily embedd the necessary dependency meta info in ELF files as a note or separate section that tools such as readelf/ldd could process. I proposed this before, for the consumption of initrd generators and automatic dep generates for rpm/dpkg, but there was no interest from anyone else to get consume this data, so I dropped it. If people are interested in it again, we can certainly revive the discussion.

IIRC MacOS has weak library deps, it would be great if we had the same on Linux/ELF: i.e a way to declare that certain symbols shall be backed by a library that is laoded the moment the symbol is resolved only, and ensuring the symbol resolves gracefully to NULL if that lib cannot be found. Given we don't have that, we just build the concept manually via dlopen() right now.

[poettering]

Sorry, but no. Splitting this up makes a mess, since it makes sharing internal code much harder. You either have to make all our internal helpers public symbols (which means namespacing issues, ABI stability and all that fuck), or you "statically" compile the shared libraries, i.e. you duplicate the internal helpers in each split up library, exploding the size on disk. Which is also terrible.

BTW, the sharing of the code in systemd is absolutely crucial. There's a reason why systemd's source code footprint is actually relatively small, even though we have a so many binaries these days. The whole of systemd is only a bit larger (15%?) than wpa_supplicant for example, and half the size of glibc for example.

[poettering]

Shouldn't libsystemd be entirely eliminated and replaced by APIs that go through IPC? Remember, if you've found one security lapse, there are ten you haven't found yet.

libsystemd contains at least three APIs that are just the client-side for IPC:

1. sd-bus as API for D-Bus
2. sd_notify() as API for systemd's simple ready notification protocol
3. sd_journal_print() as API for structured logging to journald

It doesn't really make sense to have an IPC API for these. I mean, if you then use these APIs to access these IPC APIs you add an eternal loop of IPC ;-)

[Lastique]

Sorry, but no. Splitting this up makes a mess, since it makes sharing internal code much harder.

Why does it make sharing code harder? You don't have to make your private APIs public, or indeed even include in the public documentation. You could even separate the client-side APIs from your internal APIs into different libraries. You note yourself there are a number of client-oriented APIs:

libsystemd contains at least three APIs that are just the client-side for IPC:

1. sd-bus as API for D-Bus
2. sd_notify() as API for systemd's simple ready notification protocol
3. sd_journal_print() as API for structured logging to journald

I'm not sure about D-Bus, but extracting the latter two into a separate library should be rather painless and leave the rest of the libsystemd pretty much intact.

[poettering]

Why does it make sharing code harder? You don't have to make your private APIs public, or indeed even include in the public documentation. You could even separate the client-side APIs from your internal APIs into different libraries. You note yourself there are a number of client-oriented APIs:

See and I thought I explained this in the very next sentences which you truncated. let me try with another angle: ELF symbols are either exported (and thus usable by other ELF modules, but pollute the global symbol namespace, and need at least some consideration on ABI/API stability) or not (and thus only accessible to the module they are defined in). There is no concept in ELF for exporting symbols only towards some modules that are dynamically linked together and not to others. So, if you have a helper func you want to use across our codebase, what would you mark it now? Currently it's easy, they are all private, and because libsystemd.so is one thing only it's straightforward. If you now want to share them among many libs, you must make them public, and then fuck me.

So no. Nothing of this is "rather painless" or "easy". Developing libraries is hard. And needs a lot of thinking about the future, because so much stuff trickles down into ABIs/APIs if you are not careful right at the beginning, and keep your options open.

[poettering]

I mean, there's a reason why even glibc is trying very hard to merge librt/libdl/libpthread and so on back into libc: it's an unmaintainable mess to pretend these are separate in their implementation even if they really really aren't.

[Lastique]

Exporting a symbol from a library does not automatically make that symbol part of the public API. The way I see it, it doesn't matter which symbols you export (apart from the public API), and you don't have to maintain any sort of compatibility wrt. symbols that are not part of the public API.

[Lastique]

I mean, there's a reason why even glibc is trying very hard to merge librt/libdl/libpthread and so on back into libc: it's an unmaintainable mess to pretend these are separate in their implementation even if they really really aren't.

glibc is merging because there is no reason to split it, and because for most users these libraries are always used together. libsystemd is different. There is a very good reason to split it - because it would reduce external dependencies and the amount of code used by clients, and because parts of libsystemd used by systemd services are not likely to be needed by other clients.

[Conan-Kudo]

Yeah, even I don't really buy that argument and I'm the one asking for the library split. It's reasonable to indicate that doing so would increase the exposure to unstable symbols and that would be a problem. There are a few ways to hide symbols through GCC, but if you need to do cross linking privately, that would break it.

[poettering]

Exporting a symbol from a library does not automatically make that symbol part of the public API. The way I see it, it doesn't matter which symbols you export (apart from the public API), and you don't have to maintain any sort of compatibility wrt. symbols that are not part of the public API.

Well, it's not your choice to make. ELF linking works the way it works. C works the way it works. It's not a decision of philosophy, it's how this is technically implemented.

I mean, maybe if it makes this easier to understand think about the namespacing issue alone: in C libraries you have to prefix your functions with something (unless you pull a POSIX and say "I am god, I don't need to prefix my stuff!", but even they departed from that model, and nowadays have prefixes such as posix or pthread). Thus if we'd suddenly export all our internal symbols, then you'd have to at the very least prefix them all with sd_ or so.

(and add versioning and so on, but jeezuz fuckin christ, yukk, don't make me)

[poettering]

I have the suspicion that the folks arguing for the split up never tried of maintain a set of somewhat related C libraries with 100% API compat over a decade, themselves. If they just tried, they'd soon realize what a mess that will become for them.

[Lastique]

Yes, for sure, add an sd_ prefix to names to avoid name clashing. You can even use a special prefix for your internal names. But you don't have to add versioning to your internal symbols since noone but systemd itself should be using them, in a version-synchronized manner. I don't see adding a prefix as something that makes maintenance or code sharing much harder.

[poettering]

Also, let's not forget one more thing: dynamic symbol resolving is – to this day – something computers aren't really that great at. It's slow and it's problematic for security. I think exporting relevant API calls is important and has many valid usecases, hence we do it. But maybe let's not totally ignore the fact that if start stuffing export tables with tons of internal symbols this just turns everything into shit.

[ericcurtin]

it could help minimize initrd size (as you can exclude the libraries that will never be dlopened)

How initrd generator is supposed to know which libraries will be needed?

We would have to manually specify the optional dependencies.

[bluca]

Yes, for sure, add an sd_ prefix to names to avoid name clashing. You can even use a special prefix for your internal names. But you don't have to add versioning to your internal symbols since noone but systemd itself should be using them, in a version-synchronized manner. I don't see adding a prefix as something that makes maintenance or code sharing much harder.

As already mentioned: there is no point in going in circles. The answer is no, we are not splitting anything, and no amount of back and forth will change that. Sorry if it's not the answer you were hoping for, but let's stop here, and focus keep this ticket on the topic of what we want to do: document dlopen, remove gcrypt and cap2 linking, provide a notify self contained example

[poettering]

We would have to manually specify the optional dependencies.

So yes, that's what everyone (for example Dracut) already does, in particular for things like NSS/PAM, but also for systemd's various pre-existing dlopen() dep.

As mentioned several times before: if there's interest by intird generator folks and package mgmt folks we'd be happy to declare the dlopen() deps in some ELF section or note, so that they can programmatically derive them from our binaries.

But every time I bring this up, the responses are lukewarm at best...

[ericcurtin]

IIRC MacOS has weak library deps, it would be great if we had the same on Linux/ELF: i.e a way to declare that certain symbols shall be backed by a library that is laoded the moment the symbol is resolved only, and ensuring the symbol resolves gracefully to NULL if that lib cannot be found. Given we don't have that, we just build the concept manually via dlopen() right now.

Support adding this to Linux/ELF :+1: Would be a neat feature...