CachyOS kernels have poor perf compared to Arch's for cosmic-DE on vanilla arch

[Quackdoc]

I am using vanilla arch but was told to report these anyways.

Originally I was testing cachyOS sched-ext kernels but found I had some really bad performance using them, I wound up downgrading to the "vanilla" sched-ext kernel and found the issues went away.

After that I compared linux-cachyos-eevdf-6.8.6 to arch's vanilla kernel and found the same issues.

Some times I gathered

Cosmic-comp -> cosmic-greeter loadtimes; Vanilla: 6s Cachy: 40s (and then needed systemctl restart) Cosmic-session fill load; Vanilla: 10s Cachy: 1min 10s (Cosmic-panel half broke and started spewing a bunch of dbus errors into journalctl)

Some other miscellaneous issues I found that cachyOS kernel with cosmic-comp stuttered really badly.

Neither journalctl nor dmesg had any pertinent logs, but I can gather them if necessary. Any other logging source I can also grab. I'm not really in a state where I can compile my own kernel unfortunately, I neither have the time nor hdd space.

[sirlucjan]

Could you provide a little information about your hardware?

[Quackdoc]

CPU: Ryzen 2600 RAM: 16gb ddr4 GPU: Arc A380 Storage: Root storage F2FS, Secondary Storage bcachefs (nothing loaded on it, issues occur even when unmounted)

any other info that would be helpful?

[sirlucjan]

It is possible that this is related to amd-pstate. @ptr1337 maybe you have a suggestion?

[Quackdoc]

this is the output of cpupower for me, I can set amd-pstate if it would help

cpupower frequency-info
analyzing CPU 0:
  driver: acpi-cpufreq
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 1.55 GHz - 3.40 GHz
  available frequency steps:  3.40 GHz, 2.80 GHz, 1.55 GHz
  available cpufreq governors: conservative ondemand userspace powersave performance schedutil
  current policy: frequency should be within 1.55 GHz and 3.40 GHz.
                  The governor "schedutil" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 3.57 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: no

EDIT: does my CPU even support it? docs say the belowl my CPU is a Zen+ not Zen2

Currently, some of the Zen2 and Zen3 processors support amd-pstate. In the future, it will be supported on more and more AMD processors.

[ptr1337]

Is this also present on other DEs?

[Quackdoc]

I tested on sway, sway seems to not really present these issues. Cosmic application on top of sway do (and even crashed sway multiple times). but general use seems more or less fine under idle.

I did however experience severe performance degradation in games. DXVK games would have on average lower performance, and when shaders were being compiled the games stuttered way worse on cachyOS kernels. but even without shader compilation I notice drops down to 30fps regularly.

So issues do exist, just not as severe on cosmic. I have not tested KDE or Gnome, but can if it would help.

[Quackdoc]

I should be able to compile a kernel sometime soon, is https://github.com/CachyOS/kernel-patches/blob/master/6.8/all/0001-cachyos-base-all.patch the only relevant patch file? if so I can probably compile a kernel with each of these patches one at a time to locate the issue.

[ptr1337]

I should be able to compile a kernel sometime soon, is https://github.com/CachyOS/kernel-patches/blob/master/6.8/all/0001-cachyos-base-all.patch the only relevant patch file? if so I can probably compile a kernel with each of these patches one at a time to locate the issue.

Actually you would need to create a own patch. Just reverting the complete patch does not help much. I'd suggest you to remove following patch: https://github.com/CachyOS/kernel-patches/blob/master/6.9/0004-cachy.patch

and check if it improves. You can do this simply doing this like that:

git clone https://github.com/cachyos/kernel-patches.git
cd kernel-patches/6.8
rm -rf 0001-aes-xts.patch 0004-cachy.patch 
# After that cat all patches into one big one
cat *.patch > 0001-cachyos-base-all.patch

and then replace the 0001-cachyos-base-all.patch into the PKGBUILD directory. Thanks for helping to bisect it, we didnt get such a report, yet.

[Quackdoc]

I do believe the issue to be the 6.8/0005-cachy.patch (testing on kernel 6.8 because 6.9 has breaking bugs with bcachefs) I've done a dirty kernel compile removing only it, and the issue doesn't seem to present itself, i'm currently doing a clean compile with only that patch to see if the problem is replicated. It seems like a rather large patch so im not sure where to get started looking into it.

EDIT: yup I have confirmed that it is the issue. Any idea where I should start looking? Im not super familiar with kernel work.

[Quackdoc]

I went ahead and built the scx kernel without the cachyos patch file. Under normal operation the kernel is fine, but when I load an scx_scheduler performance of the scheduler is far inferior to the 6.7 kernel from the cachy repos. I haven't yet built 6.8 vanilla with scx patches and cachyos doesn't seem to have it built either yet so I will try and test it in the future when I get time

[ptr1337]

Please test the latest linux-cachyos-rc.

[Quackdoc]

testing 6.9.0-rc6-4-cachyos-rc It seems like the issues are mostly resolved with this. but not 100%

cosmic-greeter seems to still take a long time to load, but inside once cosmic-session loads it is a lot better, cosmic applications load faster. but I am experiencing a variety of other issues I don't know if are related to the patch. Somethings like games are taking much longer to load (wine is taking ages). Firefox has crashed twice while typing this and a few others.

Ill test a vanilla linux RC build later, but so far it seems like at least some of the problems are resolved

[ptr1337]

Could you recheck with the final build? (6.9.0-3) Weve dropped most amd-pstate patches, that should fix it. Actually there are a bunch of isuses with Zen 3 and lower.

[xz-dev]

I'm not see 9.6.0-3. is it 6.9.0-2 ?

[Quackdoc]

it is loads better, but the cachy patch is still causing some issues.

With the patch, most things load fine, cosmic-greeter still takes a long time to load, but every other cosmic* app seems to work fine. the patches however do cause the compositor to "freeze" for about 2 or 3 seconds relatively occasionally. The games I played also still have some perf issues. the game I tested, genshin jumped between 40-54fps when the kernel was compiled with 0005-cachy.patch

When I recompiled the kernel without said patch all my perf issues went away, Solid 60fps, cosmic greeter loads fast, cosmic-comp doesnt hang up at all etc.

so there are still some issues with the cachy.patch. I did take a look at it and ill try to break it unto various chunks when I get the chance, but im not too familiar with kernel stuff anymore, at a glance some of the stuff that could potentially be effecting it that I can see are

• intel nvme
• Compile settings tweaks
• timer tweaks
• sched/fair.c changes (tested scx kernel, would any of the scx schedulers be able to work around this stuff?)
• page alloc?
• lru gen
• evdev

if there is anything else I missed I can take a look at it, but this may take me a while since I don't know what is dependant on what

[ventureoo]

it is loads better, but the cachy patch is still causing some issues.

With the patch, most things load fine, cosmic-greeter still takes a long time to load, but every other cosmic* app seems to work fine. the patches however do cause the compositor to "freeze" for about 2 or 3 seconds relatively occasionally. The games I played also still have some perf issues. the game I tested, genshin jumped between 40-54fps when the kernel was compiled with 0005-cachy.patch

When I recompiled the kernel without said patch all my perf issues went away, Solid 60fps, cosmic greeter loads fast, cosmic-comp doesnt hang up at all etc.

so there are still some issues with the cachy.patch. I did take a look at it and ill try to break it unto various chunks when I get the chance, but im not too familiar with kernel stuff anymore, at a glance some of the stuff that could potentially be effecting it that I can see are

• intel nvme
• Compile settings tweaks
• timer tweaks
• sched/fair.c changes (tested scx kernel, would any of the scx schedulers be able to work around this stuff?)
• page alloc?
• lru gen
• evdev

if there is anything else I missed I can take a look at it, but this may take me a while since I don't know what is dependant on what

Can you test linux-zen as well? Because many of tweaks we apply in 0005-cachy.patch are taken from zen kernel, and maybe some of them cause regressions. Besides that, I don't see any other reason.

[ptr1337]

I'm not see 9.6.0-3. is it 6.9.0-2 ?

We need to push into the cachyos-v3 and cachyos-v4 repository the kernel all time a pkgrel higher, to avoid a module mismatch at installation time.

[Quackdoc]

the latest linux-zen kernel does not have these issues within an hour of testing but I can keep testing if needed

[ptr1337]

the latest linux-zen kernel does not have these issues within an hour of testing but I can keep testing if needed

Did you test the above mentioned linux-cachyos? If its still present, then only bisecting will help.

[Quackdoc]

I built the right one, When I have time I'll look at the patchset when I can and try to break it up. and test each change seperately

[ptr1337]

@Quackdoc is this still a thing or is it nowdays fine?

[ptr1337]

Since no response, closing here for now.

Reopen if its still an issue.

[Quackdoc]

sorry for the extremely late reply, I've been low on spare time lately, issue is still occurring but I'm not sure how much time I have to invest in debugging. is there a split up patch for cachy.patch part where I can test individual aspects of it? it doesn't look like i'll have time to manually go through it