Improve linux OS detection

[michalnicp]

Goal

Our current method for detecting whether a host is linux, macos, or windows is crude and relies on maintaining a list of linux distros. As seen in the following issues #8433, #8534, #6760 we obviously don't have a complete list.

We should come up with a better approach that is future proof doesn't rely on a manually maintained list.

How?

• Add a new field to hosts eg os_type that is either windows, linux or macos.
• populate os_type using the build_platform column from osquery_info table

[jcgruenhage]

build_platform doesn't really work when cross compiling for a different platform though, right? It'll contain the wrong platform in those cases.

Maybe it would make sense to make platforms more configurable, so that users could also add their own platforms (what about BSDs for example?), and for each platform we could have a query that determines whether something matches this platform (for the hosts I'm interested in, it would be select 1 from os_version where platform like "debian" or platform like "void"; The most important platforms and distributions could be in a default here, so it works out of the box for most users, and those with weird distributions can override it locally without having to contribute that upstream to you.

[jcgruenhage]

Also, considering this is a bigger discussion and might take a while: I hope this doesn't block the tiny MR that is #8534? Because I'd like to see that in the next release, as we're preparing to use fleet at work, and without it being able to detect the right platform, the main feature we want to use (policies) isn't available for the devs that might use exotic distros on their work laptops.

[jcgruenhage]

Another thing I noticed: If I filter by platform on the hosts tab, I do already see the Void machines under the Linux platform, which makes it relatively confusing for users why the policies for a Linux host aren't evaluated against these hosts, when they show up under hosts when filtering for Linux.

[chiiph]

Hi @jcgruenhage, really sorry this is causing you issues, we are having a conversation internally to come up with a strategy for cases like this since sadly it's not always as simple as adding a string to a slice. In particular for void, I'm not sure software inventory would work as it is right now, for instance.

As soon as we define the strategy we'll take, we'll update this issue.

[jcgruenhage]

Oh no, it's most certainly not as easy as adding a string to a slice, didn't want to imply that. It does mean though, that a few other features do end up working, especially policies, which is the thing we're most interested in. Just because Void is in that list, it doesn't mean that Void is a fully supported target. If you're not willing to merge that change though, which I could fully understand, we'll find another way. Probably a private fork, which includes those changes, until a proper solution is found and released later on.

As for software inventory, I'm not quite sure how that currently does work. I assume there are built-in queries that list the software and their versions, which rely on the different tables for packages that are exposed by osquery (and potentially extensions there). If I want to help out here, would that have to happen via contributing to osquery upstream, or would it maybe make more sense to write osquery extensions for this? I am fairly new to this whole stack and can't really tell how exactly the pieces all fit together.

[chiiph]

Hi there! Sorry for the delay. We discussed internally the topic of how to approach these situations where a linux distro that is new to Fleet is needed. The decision we've come to is the following: given time constraints, we are only going to officially support CentOS, Redhat, Amazon Linux, Debian, and Ubuntu. We will review and accept PRs to add support for other distros, but we won't make any assurances as to how well they will behave in Fleet.

So for instance, the PR that adds void to the list of distros will be reviewed and very likely merged, but we simply can't prioritize the work that needs to go in for making sure other features such as software inventory work well. You're more than welcome to contribute PRs for it, though, we'll do our best to keep up with their review and release.

For adding software inventory to void, assuming it doesn't already work, osquery would have to understand how to list software for that distro. If it's RPM or DEB based, that's already covered, but if it's another package manager, you need to extend the query for linux in Fleet to also UNION that table, and if that is not covered in osquery, then you would need to implement the table in osquery.

Does that make sense?

[jcgruenhage]

Hi there! Sorry for the delay. We discussed internally the topic of how to approach these situations where a linux distro that is new to Fleet is needed. The decision we've come to is the following: given time constraints, we are only going to officially support CentOS, Redhat, Amazon Linux, Debian, and Ubuntu. We will review and accept PRs to add support for other distros, but we won't make any assurances as to how well they will behave in Fleet.

Sounds like a very reasonable approach. Thanks for being open to contributions that add improve support for distributions you don't want to (or rather, aren't able to because of time constraints) officially support.

So for instance, the PR that adds void to the list of distros will be reviewed and very likely merged, but we simply can't prioritize the work that needs to go in for making sure other features such as software inventory work well. You're more than welcome to contribute PRs for it, though, we'll do our best to keep up with their review and release.

This is perfectly fine, and I'd never expect more than this. Working on supporting niche distros just simply doesn't make sense for you, and that's okay.

For adding software inventory to void, assuming it doesn't already work, osquery would have to understand how to list software for that distro. If it's RPM or DEB based, that's already covered, but if it's another package manager, you need to extend the query for linux in Fleet to also UNION that table, and if that is not covered in osquery, then you would need to implement the table in osquery.

osquery doesn't have support for listing software on Void Linux. Void is using XBPS, for which osquery doesn't have a table yet. I've started implementing an extension that exposes a table for xbps packages based on https://github.com/polarlabs/osquery-rust (or rather my fork of it, considering it wasn't quite working out of the box). While the osquery source code is quite intimidating, the libraries out there for implementing extensions are fairly nice to use, and I'm looking forward to do more of that. Would queries that rely on such extensions be an option for fleet?

[chiiph]

Would queries that rely on such extensions be an option for fleet?

That's a good question. We have queries that depend on extensions, but they are extensions that we are maintaining. The way we do this is by using discovery queries, but for software, it'd be tricky cause we rely on multiple tables.

You could add it as a separate query and ingest it separately, though. That would be a bit more involved in terms of code, but it shouldn't be different than the patterns we already have.

[jcgruenhage]

I'll try looking into the fleet side of things later on. The extension is working properly now, if anyone is interested feel free to take a look over at https://git.jcg.re/jcgruenhage/osquery-voidlinux/src/branch/main/src/main.rs. Except for various side-quests in the libraries involved, this was surprisingly straight-forward.