Adopt "clear linux" patches?

[colemickens]

Has anyone spent much time looking at Clear Linux and the patches they carry that get them such noticeable perf bumps?

I'd be curious to see if we could (optionally) support some of those patches in nixpkgs?

[jcumming]

There is a lot of valuable technology in Clear Linux (like the auto proxy stuff) that I'd love to see in NixOS.

One of the reasons Clear Linux is so fast is that the Clear Linux team at Intel does tons of profiling and optimization.

[matthewbauer]

I think having a NixOS profile or module to do this would be very useful! it would be interesting to see how close to Clear Linux performance we can get.

/cc @volth

I don't think it's a good idea to make this the default, however. There is always a good reason these things are not made the default for GCC, Linux, and other software. For instance, Clear Linux only supports Westmere and later Intel CPUs, as well as disabling. Based on the Phoronix article (https://www.phoronix.com/scan.php?page=article&item=ubuntu-clear-tweaks&num=1) and https://github.com/InBetweenNames/gentooLTO, here are some of the flags that could be useful:

GCC

arch: westmere tune: haswell configure flags:

--enable-offload-targets=nvptx-none
--disable-libunwind-exceptions
--enable-lto

(possible) C flags:

-O3
-feliminate-unused-debug-types
--param=ssp-buffer-size=32
-Wl--copy-dt-needed-entries-m64-fasynchronous-unwind-tables
-fasynchronous-unwind-tables
-fno-semantic-interposition
-ffat-lto-objects
-fno-signed-zeros
-fno-trapping-math
-fassociative-math
-fexceptions
-ftree-loop-distribute-patterns
-Wl,-sort-common
-fno-semantic-interposition
-fipa-pta
-fdevirtualize-at-ltrans

Glibc

--enable-tunables=...

Linux

patches: https://github.com/clearlinux-pkgs/linux

Systemd

patches: https://github.com/clearlinux-pkgs/systemd

[davidak]

Do we have some sort of independent benchmark suite?

@volth i packaged phoronix-test-suite recently. https://github.com/NixOS/nixpkgs/pull/62509

[vcunat]

I suppose the discussion should be more about the individual changes, probably in separate threads/tickets. Generally though, if some patches/settings are so good, why don't they get integrated/defaulted upstream?

GCC:

• IIRC system-wide -O3 is not recommended, as it generally allows expensive trade-offs that you probably only want to take on performance-critical parts (and still maybe not all of them)
• arch, tune: these are about improving performance on some CPUs at the expense of making others slower or not working at all – that doesn't seem suitable for general-purpose distribution/binaries (general-purpose in terms of CPU choice). Adding yet another "platform" (in terms of rebuilding and distributing) would be quite expensive, so the best I expect is for interested people to build everything themselves, but then they might prefer even closer tailoring, e.g. -march=native.
• LTO: opensuse now enables this system-wide and only disables it on particular packages, so that sounds OK, but we should first upgrade the default compiler to 8 or 9

[OpenSourceAnarchist]

@vcunat I agree that creating multiple platforms for binary redistribution would be inefficient and having users build packages themselves is a much better solution. As a previous GentooLTO user, I would LOVE to have an easy way to build everything with -march=native -O3 (or -O2 -ftree-vectorize) LTO, and graphite optimizations. Then, being able to override individual packages with "toned down" flags if necessary. This is a very Gentoo-y thing and there isn't a reason to not allow NixOS users to do the same thing if desired, so long as they accept the consequences and additional work involved.

I would also love to see LTO enabled tree-wide after a GCC/LLVM upgrade. Theoretically, every package should work with it, so finding packages and reporting bugs upstream would be extremely beneficial.

I'm being completely honest here: NixOS could capture a huge percentage of Gentoo users by allowing users to have these features (setting custom CFLAGS/CXXFLAGS/LDFLAGS and overriding for packages). I'm falling in love with NixOS and am willing to give up Gentoo for it, but I miss the simple MAKEOPTS configuration. There is a whole group of people who will do anything for an extra 1% performance gain, even if it is a placebo effect!

[vcunat]

Well, -march is now configurable (in a pure way, of course), so you might push towards adding more such options. I don't think it's difficult to add. But beware, nix will force way more recompilation on you than gentoo did. (Per-package CFLAGS/... overrides seem rather easy already.)

[OpenSourceAnarchist]

@volth That is quite a dirty way of doing it :) I might try a test config with that later...

@vcunat I'm talking about something like this specifically, and in general, all of these files/workarounds used by GentooLTO. Having overlays like this becomes extremely difficult to maintain in Nix (basically a separate overlay for every package with a different stdenv with workarounds for each of these).

If you're up for a radical suggestion, allow users to specify a global (and per-package if desired) list of C/CXX/LDFLAGS along with successive lists of FLAGS if the previous one fails. This is something that GentooLTO does not do that would make such a project so much easier to declare in one file.

Essentially, if -march=skylake -O3 -flto fails to compile/pass tests, then try again with -march=skylake -O3 -flto -ffat-lto-objects, then -march=skylake -O3 -fno-lto, then -march=skylake -O2 -fno-lto and so on. Literally solves 95% of the issues with local builds and aggressive optimizations. The rest can be written per-package (though some packages have their own optimizations in the build phase which could cause trouble with this method... just a suggestion!).

Edit: This is obviously impure and would need to be explicitly enabled by willing users, just like how -march=native is currently handled by a stdenv adapter.

[vcunat]

You don't really need to override stdenv for individual packages; you can e.g. override those packages directly to amend CFLAGS or NIX_CFLAGS* passed to stdenv.mkDerivation.

[OpenSourceAnarchist]

How does that work? The only documentation I found was on a similar thread on Discourse about overlays for -march=native on specific packages (https://discourse.nixos.org/t/targeting-particular-x86-64-generation-haswell-ivybridge-skylake/2280/24).

[devhell]

Can't wait for a "ricer" option. :laughing:

[OpenSourceAnarchist]

@devhell You jest, but I am very eager to switch from a riced GentooLTO install to NixOS when it becomes easy to build all locally optimized, aggressively compiled packages :)

[davidak]

@OpenSourceAnarchist that sounds interesting. are there any studies or benchmarks how much you can improve the performance with that on a recent CPU? Can you estimate how many % it can bring?

[OpenSourceAnarchist]

@davidak It is definitely an interesting project that, on average, tends to be roughly 10% faster overall (compared to standard -O2 compilation). In select workloads, an optimized setup will yield a nearly 200% faster package on modern CPUs compared to -O2. Of course, compiling from source takes some time, though an entire world rebuild on my laptop (Dell Latitude 5300) easily finishes overnight.

Stats can be seen (with a grain of salt since dependent libraries aren't compiled with these flags) here: https://www.phoronix.com/scan.php?page=article&item=gcc9-skylake-avx512

Some packages work OOTB with aggressive optimizations, others need workarounds (see GentooLTO's work here: https://github.com/InBetweenNames/gentooLTO/blob/master/sys-config/ltoize/files/package.cflags/ltoworkarounds.conf). All in all, I'd love to see NixOS support more aggressive optimizations (with workarounds added as-needed) for those who are willing to take the time and risk of using them. Obviously it would have to be an exclusively opt-in feature since it would most likely violate purity and could break after any given update. Though the atomic feature of NixOS is what makes me want to switch to it from Gentoo. If a package update breaks on Gentoo, you have a non-functional system until you identify what compiler flag caused an issue. Even worse, a build can succeed but the package can be broken, which takes a lot of troubleshooting (was it a dependent library, an optional dependency, etc?). On NixOS, I can rollback and test different combinations until everything works. Absolutely ingenious system that any source-based distro should adopt.

[matthewbauer]

Some packages work OOTB with aggressive optimizations, others need workarounds (see GentooLTO's work here: https://github.com/InBetweenNames/gentooLTO/blob/master/sys-config/ltoize/files/package.cflags/ltoworkarounds.conf). All in all, I'd love to see NixOS support more aggressive optimizations (with workarounds added as-needed) for those who are willing to take the time and risk of using them. Obviously it would have to be an exclusively opt-in feature since it would most likely violate purity and could break after any given update.

Those workarounds are pretty extensive though! I think we can accomplish those with overlays, although it's a little scary how much can break from LTO. Maybe an external project could try those out.

[OpenSourceAnarchist]

It's unfortunate that LTO (and even -O3) breaks so many packages. It shouldn't by design, but projects like these help upstream find bugs so less workarounds are needed in the future.

I'll be willing to test and contribute bug reports to any "NixLTO" efforts :laughing:. I'm still learning Nix and NixOS in general, but if this gets any traction I'll switch over to NixOS full-time and contribute workarounds through overlays that I find. Definitely could convince some GentooLTO folks to try out NixOS too for this project.

In fact, testing packages for build failures would be way easier with NixOS compared to Gentoo!

[mdedetrich]

It's unfortunate that LTO (and even -O3) breaks so many packages. It shouldn't by design, but projects like these help upstream find bugs so less workarounds are needed in the future.

Does it break so many packages though? Clear Linux is already having a large package repository and it doesn't appear to be causing that many issues.

I agree though that the sooner we find the breakages with LTO/O3 the sooner they can be fixed. Nixos already has great infrastructure in place to support patching of packages anyways.

Personally I would like to see a distribution of NixOS with the same flags/adjustments that Clear Linux (or similar) uses. This of course would be opt in however in my eyes its well worth the effort considering the majority of desktop/laptop users would benefit from such a patchset.

[vcunat]

No, LTO should break only a relatively small fraction of packages, and it's an option (in principle) suitable for enabling by default. I already posted this SUSE link; I expect we'd be able to piggy-back on their work on fixing some of the breakages, too.

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[Atemu]

Still important to me.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/cirnos-brainstorming-period/10056/10

[stale[bot]]

I marked this as stale due to inactivity. → More info

[Atemu]

Crazy idea: Periodically assimilate the clear-linux optimisation declarations and add an attr to mkDerivation that lets you delare a clear-linux package name to automatically apply the optimisations (clflags, patches etc.) of.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-benchmark/16225/3

[Lillecarl]

https://xanmod.org/ Here's a kernel currently being built for NixOS with some clearlinux patches and other goodies. I intend to test Anbox with this kernel soon-ish

boot.kernelPackages = with pkgs.linuxKernel.packages; linux_xanmod;

[WillPower3309]

https://xanmod.org/ Here's a kernel currently being built for NixOS with some clearlinux patches and other goodies. I intend to test Anbox with this kernel soon-ish

boot.kernelPackages = with pkgs.linuxKernel.packages; linux_xanmod;

Small nitpick but the expression can be slightly simplified to boot.kernelPackages = pkgs.linuxPackages_xanmod; I've been using it with great success! :)

[Lillecarl]

https://xanmod.org/ Here's a kernel currently being built for NixOS with some clearlinux patches and other goodies. I intend to test Anbox with this kernel soon-ish boot.kernelPackages = with pkgs.linuxKernel.packages; linux_xanmod;

Small nitpick but the expression can be slightly simplified to boot.kernelPackages = pkgs.linuxPackages_xanmod; I've been using it with great success! :)

Sure, but then you can't find it by searching on https://search.nixos.org which is something i value higher than character count.

[WillPower3309]

Does Intel provide a way for users to see what compilation flags they use on the clear Linux packages? Phoronix benchmarks always list them yet I haven't been able to find them myself. I'm thinking if they can be verified via benchmarking to provide a significant performance increase, I'd make some PRs to nixpkgs implementating the more performant flags.

[Atemu]

You can simply have a look at their sources; they've got packaging scripts.

[GrabbenD]

Any updates to this? I'm new to NixOS and I'm unsure how to achieve this in a clean way

[gytis-ivaskevicius]

low hanging fruit to increase performance would be to provide targets for x86_64-v3/x86_64-v4 (not related to clear linux patches, only related to improved performance)

[SuperSamus]

202526

[blitz]

If anyone is interested in building something that benchmarks the NixOS Linux kernel, I'd love to chat. I'm @js:ukvly.org on Matrix.

I think some benchmarking infrastructure would be a prerequisite for performance optimizations.

[ShalokShalom]

Why dont we just outright build the Clear Linux kernel? Arch is doing that, and I raised an issue above this very comment. Feel free to comment there. 🙂

[GrabbenD]

Why dont we just outright build the Clear Linux kernel?

Xanmod kernel has patches from Clear Linux and it's available as a NixOS package.

However, most of Clear Linux's performance isn't coming from the kernel but rather how the packages are compiled and various performance patches: https://github.com/clearlinux-pkgs/linux

---

Applying patches to kernel+package shouldn't be too hard.. However, if you really want to have the same level of performance as Clear Linux you'd also have to use the same compiler optimizations (custom flags) which isn't trivial in NixOS:

https://www.reddit.com/r/NixOS/comments/146wdfk/comment/jntesnq/

Been there, done that. It's not worth it in my experience for a minimal Sway desktop due to compilation times and bugs: https://www.reddit.com/r/NixOS/comments/13auvya/comment/jjbndb8/

It's a shame considering x86_64-v3 + LTO is deployed in other distros such as OpenSuSE Tumbleweed and CachyOS

https://www.reddit.com/r/NixOS/comments/13auvya/comment/jjbndb8/

Sure but not everything in NixOS is built from source. There's plenty of packages which unpacks pre-compiled .deb files, one example being google-chrome.

Also, it's worth to consider if the power consumption is actually worth it. Every time a package updates, everything which depends on it will be rebuilt due to the derivation getting a new hash which triggers its dependencies/build inputs to rebuild as well to ensure reproducibility.

Furthermore you can't actually use CCache system wide (yet and until foreseeable future). However, depending on how a derivation is packed, it might be possible to specify individual packages which should use the cache: https://github.com/NixOS/nixpkgs/issues/227940#issuecomment-1546660889

Also, just heads up. Chromium based packages don't play nicely with CCache if you end up specifying it manually. On my 5950X 16 core system, Chromiun build takes 1.5 - 2 hours and you have to restart the build from the beginning if it fails as there's no caching of build layers in NixOS.

I've been down this rabbithole before and it's just not worth it from my experience. Looking back, I'd rather go the Gentoo route if you want this level of customizability.

Anyhow, here's how to tweak build flags if you still want to try this endeavor: https://www.reddit.com/r/NixOS/comments/yww3mr/comment/iwlxvb5/

---

Arch is doing that, and I raised an issue above this very comment.

CachyOS is doing it better 😉

[ShalokShalom]

I understand. ALHP provides an alternative repo, and includes LTO, -03 and v3. Would that be an option?

[GrabbenD]

ALHP repository isn't compatible with NixOS if that's the question

But if you're wondering if ALHP is an alternative to CachyOS then yes. You can actually use CachyOS repositories in a existing Arch Linux installation (check their wiki) and it's even possible to use both ALHP and CachyOS simultaneously (by placing one above the other to select priority in /etc/pacman.conf and leaving Arch Linux repositories at the bottom as fallback). I personally prefer just using CachyOS as it offers Clear Linux optimizations, LTO, O3, v4/v3/v2 and I found it to be more stable / less prone to breakage vs only using ALHP (but that's off-topic so let's not discuss it further 🙂).

I hacked together a OSTree setup using Dockerfiles meaning I can have a imperative system, rollbacks AND optimized packages :heart:

[ShalokShalom]

I was wondering if the approach - providing an entire repository - would work for us.

[Atemu]

@ShalokShalom we don't need entire new repos or infrastructure, we already have all of that. The real meat of this is getting individual patches and tweaks into individual packages. With Nix we can hide all of them by default behind a opt-in override flag anyways (or perhaps even a config attribute).

[ShalokShalom]

So what is missing? Can you link me to the according issue or PR?

[mikunimaru]

There is a lack of documentation on how to add build flags to most packages (including the kernel). Without this, other users will not be able to experiment with the build.

[06kellyjac]

https://nixos.wiki/wiki/Build_flags

https://nixos.wiki/wiki/Linux_kernel

https://nixos.org/guides/nix-pills/nixpkgs-overriding-packages.html

https://ryantm.github.io/nixpkgs/using/overrides/

[mikunimaru]

The documentation is very scattered. For example, it is not possible to understand "How to build a kernel with lto using clang'' on a single page. This is because although the method for building a kernel with clang is posted, the method for passing optimization flags when building the kernel is not posted on the same page.

[WillPower3309]

So what is missing? Can you link me to the according issue or PR?

There's nothing missing to get this implemented, nixpkgs already supports adding patches and defining flags. Unless we want an issue tracking every individual package it's just up to the community to (optionally) add those patches and flags to packages that interest them.

[ShalokShalom]

There is a lack of documentation on how to add build flags to most packages (including the kernel). Without this, other users will not be able to experiment with the build.

I dont understand. Why not add complete packages with the new binaries?

[WillPower3309]

There is a lack of documentation on how to add build flags to most packages (including the kernel). Without this, other users will not be able to experiment with the build.

I dont understand. Why not add complete packages with the new binaries?

Because nix is a source based Linux package manager. Build flags are added as default values in a package definition that can be optionally overriden by a user, so making a new package definition for a build flag that can be optional is an antipattern.

[ShalokShalom]

Oh, I see. Yeah, then there is definitely documentation missing.

[omar391]

I am new to Nixos. So maybe a naive question:

how about we have some sort of profile to get a clear Linux repo to download the package/source including compiler configs instead of the default Nix repo? That would save us the hassle of configuring individual pkg to achieve the best performance for Intel hardware.

[blitz]

While I like a well optimized system like the next person, I personally would like to see some easy to run benchmarks for NixOS. Ideally, something we can plot over time. Otherwise, we will just cargo cult compiler flags...

Is there already a NixOS performance chat somewhere where some ideas can be discussed? Maybe that's the first step to get going. This is also related to #202526.