Steam and Wine.

[Blackrucat]

I am having issues with the latest Liquorix kernel, steam is lagging and all steam games are lagging, crashing(not just the game but my entire system), not working well.

I thought it was wine or pulse problem, then I tried many things, I even reinstalled my distro (Pop_OS!), and the problem was the kernel, now I am using a stable 5.8 Linux generic kernel and steam it's working great.

[damentz]

Ok, we're missing a lot of information here. Can you provide:

• Output of inxi -b (while running Liquorix)
• Version of proton you're using
• Which games are problematic
• Which version of Pop OS you're running (may be included in inxi -b output)

Thanks

[Blackrucat]

System information

System:    Host: pop-os Kernel: 5.8.0-7642-generic x86_64 bits: 64 Desktop: N/A 
           Distro: Pop!_OS 20.10 
Machine:   Type: Desktop Mobo: First model: PTM33 v: PCB 1.X 
           serial: <superuser/root required> BIOS: Itautec ST 4160 v: ILB25IM-SL2 
           date: 12/04/2008 
CPU:       Info: Dual Core Intel Core2 Duo E7400 [MCP] speed: 1596 MHz 
           min/max: 1600/2800 MHz 
Graphics:  Device-1: AMD Turks XT [Radeon HD 6670/7670] driver: radeon v: kernel 
           Display: wayland server: X.Org 1.20.9 driver: ati,radeon 
           unloaded: fbdev,modesetting,vesa resolution: 1920x1080~60Hz 
           OpenGL: renderer: AMD TURKS (DRM 2.50.0 / 5.8.0-7642-generic LLVM 11.0.0) 
           v: 3.3 Mesa 20.3.2 
Network:   Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet driver: r8169 
Drives:    Local Storage: total: 521.66 GiB used: 74.75 GiB (14.3%) 
Info:      Processes: 221 Uptime: 7m Memory: 5.68 GiB used: 2.07 GiB (36.5%) Shell: Bash 
           inxi: 3.1.07 

PROTON: Proton-6.4-GE-1 WINE: staging

All games that run in Wine/proton aren't working well.

[damentz]

Hmm, this appears to be the issue: OpenGL: renderer: AMD TURKS (DRM 2.50.0 / 5.8.0-7642-generic LLVM 11.0.0).

Your system is using software acceleration while using Liquorix (thus the appearance of LLVM). Maybe there's some firmware missing. Does journalctl -k | grep -i firmware or sudo dmesg | grep -i firmware reveal anything interesting on your system? Otherwise, maybe the version of Mesa could be an issue.

[Blackrucat]

Same information as before, but now using Liquorix. (I gave you information, but as I said I was using 5.8.0 generic kernel(since liquorix wasn't working well on games))

inxi -b

System:    Host: pop-os Kernel: 5.11.0-8.1-liquorix-amd64 x86_64 bits: 64 Desktop: N/A 
           Distro: Pop!_OS 20.10 
Machine:   Type: Desktop Mobo: First model: PTM33 v: PCB 1.X 
           serial: <superuser/root required> BIOS: Itautec ST 4160 v: ILB25IM-SL2 
           date: 12/04/2008 
CPU:       Info: Dual Core Intel Core2 Duo E7400 [MCP] speed: 2793 MHz 
           min/max: 1600/2800 MHz 
Graphics:  Device-1: AMD Turks XT [Radeon HD 6670/7670] driver: radeon v: kernel 
           Display: wayland server: X.Org 1.20.9 driver: ati,radeon 
           unloaded: fbdev,modesetting,vesa resolution: 1920x1080~60Hz 
           OpenGL: 
           renderer: AMD TURKS (DRM 2.50.0 / 5.11.0-8.1-liquorix-amd64 LLVM 11.0.0) 
           v: 3.3 Mesa 20.3.2 
Network:   Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet driver: r8169 
Drives:    Local Storage: total: 521.66 GiB used: 75.34 GiB (14.4%) 
Info:      Processes: 226 Uptime: 2m Memory: 5.68 GiB used: 1.65 GiB (29.0%) Shell: Bash 
           inxi: 3.1.07 

journalctl -k | grep -i firmware

Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal' can see all messages.
      Pass -q to turn off this notice.

sudo dmesg | grep -i firmware

miau@pop-os:~$

gnome-firmware

[Blackrucat]

It doesn't matter If I am using Xorg or Wayland, I am having the same problem.

[damentz]

Hmm, how about: sudo dmesg | grep -P 'drm|radeon|amd'? This might tell us if anything happened with your graphics initialization that's noteworthy.

Another thing worth experimenting with, Liquorix is much newer than the PopOS kernel, so it might also behave better with a newer Mesa. You can install updated mesa and libdrm packages from here: https://launchpad.net/~oibaf/+archive/ubuntu/graphics-drivers. Simply add the repo and update your system (sudo apt-get update && sudo apt-get dist-upgrade). Just note that when you perform a system update next, you may want to remove the repository and downgrade to system packages before doing a full system upgrade next.

EDIT: Also, Mesa 21.x has a fix that makes open source drivers perform the same as mainline CFS. Here's a blog post about it from Con Kolivas: https://ck-hack.blogspot.com/2020/05/linux-56-ck2-muqss-version-02-for-linux.html

[Blackrucat]

sudo dmesg | grep -P 'drm|radeon|amd'

[    0.000000] Linux version 5.11.0-8.1-liquorix-amd64 (steven@liquorix.net) (gcc (Ubuntu 10.2.0-13ubuntu1) 10.2.0, GNU ld (GNU Binutils for Ubuntu) 2.35.1) #1 ZEN SMP PREEMPT liquorix 5.11-19ubuntu1~groovy (2021-03-20)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.11.0-8.1-liquorix-amd64 root=UUID=196d1250-0e98-4fcb-8b71-c32ff8d1402f ro quiet splash vt.handoff=7
[    0.058533] Kernel command line: audit=0 intel_pstate=disable  BOOT_IMAGE=/boot/vmlinuz-5.11.0-8.1-liquorix-amd64 root=UUID=196d1250-0e98-4fcb-8b71-c32ff8d1402f ro quiet splash vt.handoff=7
[    0.937730] usb usb1: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 ehci_hcd
[    0.948532] usb usb2: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 ehci_hcd
[    0.948999] usb usb3: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 uhci_hcd
[    0.949341] usb usb4: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 uhci_hcd
[    0.949687] usb usb5: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 uhci_hcd
[    0.950026] usb usb6: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 uhci_hcd
[    0.950392] usb usb7: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 uhci_hcd
[    0.950929] usb usb8: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 uhci_hcd
[    0.977068]     BOOT_IMAGE=/boot/vmlinuz-5.11.0-8.1-liquorix-amd64
[    1.298695] [drm] radeon kernel modesetting enabled.
[    1.298770] fb0: switching to radeondrmfb from simple
[    1.298863] radeon 0000:01:00.0: vgaarb: deactivate vga console
[    1.299246] [drm] initializing kernel modesetting (TURKS 0x1002:0x6758 0x1682:0x3180 0x00).
[    1.299512] radeon 0000:01:00.0: VRAM: 2048M 0x0000000000000000 - 0x000000007FFFFFFF (2048M used)
[    1.299515] radeon 0000:01:00.0: GTT: 1024M 0x0000000080000000 - 0x00000000BFFFFFFF
[    1.299519] [drm] Detected VRAM RAM=2048M, BAR=256M
[    1.299521] [drm] RAM width 128bits DDR
[    1.299596] [drm] radeon: 2048M of VRAM memory ready
[    1.299598] [drm] radeon: 1024M of GTT memory ready.
[    1.299610] [drm] Loading TURKS Microcode
[    1.299730] [drm] Internal thermal controller with fan control
[    1.303997] [drm] radeon: dpm initialized
[    1.304138] [drm] GART: num cpu pages 262144, num gpu pages 262144
[    1.325670] [drm] PCIE GART of 1024M enabled (table at 0x0000000000162000).
[    1.325785] radeon 0000:01:00.0: WB enabled
[    1.325789] radeon 0000:01:00.0: fence driver on ring 0 use gpu addr 0x0000000080000c00
[    1.325792] radeon 0000:01:00.0: fence driver on ring 3 use gpu addr 0x0000000080000c0c
[    1.326555] radeon 0000:01:00.0: fence driver on ring 5 use gpu addr 0x0000000000072118
[    1.326773] radeon 0000:01:00.0: radeon: MSI limited to 32-bit
[    1.326837] radeon 0000:01:00.0: radeon: using MSI.
[    1.326872] [drm] radeon: irq initialized.
[    1.343800] [drm] ring test on 0 succeeded in 2 usecs
[    1.343814] [drm] ring test on 3 succeeded in 6 usecs
[    1.520961] [drm] ring test on 5 succeeded in 2 usecs
[    1.520971] [drm] UVD initialized successfully.
[    1.521527] [drm] ib test on ring 0 succeeded in 0 usecs
[    1.521595] [drm] ib test on ring 3 succeeded in 0 usecs
[    2.187855] [drm] ib test on ring 5 succeeded
[    2.188303] [drm] Radeon Display Connectors
[    2.188305] [drm] Connector 0:
[    2.188306] [drm]   HDMI-A-1
[    2.188307] [drm]   HPD4
[    2.188308] [drm]   DDC: 0x6450 0x6450 0x6454 0x6454 0x6458 0x6458 0x645c 0x645c
[    2.188310] [drm]   Encoders:
[    2.188310] [drm]     DFP1: INTERNAL_UNIPHY2
[    2.188312] [drm] Connector 1:
[    2.188312] [drm]   DVI-I-1
[    2.188313] [drm]   HPD1
[    2.188314] [drm]   DDC: 0x6460 0x6460 0x6464 0x6464 0x6468 0x6468 0x646c 0x646c
[    2.188316] [drm]   Encoders:
[    2.188316] [drm]     DFP2: INTERNAL_UNIPHY
[    2.188317] [drm] Connector 2:
[    2.188318] [drm]   VGA-1
[    2.188319] [drm]   DDC: 0x6430 0x6430 0x6434 0x6434 0x6438 0x6438 0x643c 0x643c
[    2.188321] [drm]   Encoders:
[    2.188321] [drm]     CRT1: INTERNAL_KLDSCP_DAC1
[    2.273498] [drm] fb mappable at 0xD0363000
[    2.273503] [drm] vram apper at 0xD0000000
[    2.273504] [drm] size 8294400
[    2.273505] [drm] fb depth is 24
[    2.273506] [drm]    pitch is 7680
[    2.273592] fbcon: radeondrmfb (fb0) is primary device
[    2.273598] radeon 0000:01:00.0: [drm] fb0: radeondrmfb frame buffer device
[    2.277940] [drm] Initialized radeon 2.50.0 20080528 for 0000:01:00.0 on minor 0
[   14.914124] systemd[1]: Condition check resulted in Load Kernel Module drm being skipped.

[Blackrucat]

Hmm, how about: sudo dmesg | grep -P 'drm|radeon|amd'? This might tell us if anything happened with your graphics initialization that's noteworthy.

Another thing worth experimenting with, Liquorix is much newer than the PopOS kernel, so it might also behave better with a newer Mesa. You can install updated mesa and libdrm packages from here: https://launchpad.net/~oibaf/+archive/ubuntu/graphics-drivers. Simply add the repo and update your system (sudo apt-get update && sudo apt-get dist-upgrade). Just note that when you perform a system update next, you may want to remove the repository and downgrade to system packages before doing a full system upgrade next.

EDIT: Also, Mesa 21.x has a fix that makes open source drivers perform the same as mainline CFS. Here's a blog post about it from Con Kolivas: https://ck-hack.blogspot.com/2020/05/linux-56-ck2-muqss-version-02-for-linux.html

I did what you said, and updated mesa and libdrm.

sudo dmesg | grep -P 'drm|radeon|amd'

[    0.000000] Linux version 5.11.0-8.1-liquorix-amd64 (steven@liquorix.net) (gcc (Ubuntu 10.2.0-13ubuntu1) 10.2.0, GNU ld (GNU Binutils for Ubuntu) 2.35.1) #1 ZEN SMP PREEMPT liquorix 5.11-19ubuntu1~groovy (2021-03-20)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.11.0-8.1-liquorix-amd64 root=UUID=196d1250-0e98-4fcb-8b71-c32ff8d1402f ro quiet splash vt.handoff=7
[    0.059552] Kernel command line: audit=0 intel_pstate=disable  BOOT_IMAGE=/boot/vmlinuz-5.11.0-8.1-liquorix-amd64 root=UUID=196d1250-0e98-4fcb-8b71-c32ff8d1402f ro quiet splash vt.handoff=7
[    0.938917] usb usb1: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 ehci_hcd
[    0.949866] usb usb2: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 ehci_hcd
[    0.950300] usb usb3: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 uhci_hcd
[    0.950639] usb usb4: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 uhci_hcd
[    0.951013] usb usb5: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 uhci_hcd
[    0.951382] usb usb6: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 uhci_hcd
[    0.951904] usb usb7: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 uhci_hcd
[    0.952431] usb usb8: Manufacturer: Linux 5.11.0-8.1-liquorix-amd64 uhci_hcd
[    0.983850]     BOOT_IMAGE=/boot/vmlinuz-5.11.0-8.1-liquorix-amd64
[    1.301850] [drm] radeon kernel modesetting enabled.
[    1.301926] fb0: switching to radeondrmfb from simple
[    1.302017] radeon 0000:01:00.0: vgaarb: deactivate vga console
[    1.302416] [drm] initializing kernel modesetting (TURKS 0x1002:0x6758 0x1682:0x3180 0x00).
[    1.302673] radeon 0000:01:00.0: VRAM: 2048M 0x0000000000000000 - 0x000000007FFFFFFF (2048M used)
[    1.302676] radeon 0000:01:00.0: GTT: 1024M 0x0000000080000000 - 0x00000000BFFFFFFF
[    1.302680] [drm] Detected VRAM RAM=2048M, BAR=256M
[    1.302682] [drm] RAM width 128bits DDR
[    1.302775] [drm] radeon: 2048M of VRAM memory ready
[    1.302776] [drm] radeon: 1024M of GTT memory ready.
[    1.302785] [drm] Loading TURKS Microcode
[    1.302903] [drm] Internal thermal controller with fan control
[    1.307171] [drm] radeon: dpm initialized
[    1.307314] [drm] GART: num cpu pages 262144, num gpu pages 262144
[    1.327195] [drm] PCIE GART of 1024M enabled (table at 0x0000000000162000).
[    1.327311] radeon 0000:01:00.0: WB enabled
[    1.327315] radeon 0000:01:00.0: fence driver on ring 0 use gpu addr 0x0000000080000c00
[    1.327318] radeon 0000:01:00.0: fence driver on ring 3 use gpu addr 0x0000000080000c0c
[    1.328079] radeon 0000:01:00.0: fence driver on ring 5 use gpu addr 0x0000000000072118
[    1.328326] radeon 0000:01:00.0: radeon: MSI limited to 32-bit
[    1.328381] radeon 0000:01:00.0: radeon: using MSI.
[    1.328418] [drm] radeon: irq initialized.
[    1.345295] [drm] ring test on 0 succeeded in 2 usecs
[    1.345307] [drm] ring test on 3 succeeded in 7 usecs
[    1.522458] [drm] ring test on 5 succeeded in 2 usecs
[    1.522467] [drm] UVD initialized successfully.
[    1.523040] [drm] ib test on ring 0 succeeded in 0 usecs
[    1.523131] [drm] ib test on ring 3 succeeded in 0 usecs
[    2.188792] [drm] ib test on ring 5 succeeded
[    2.189217] [drm] Radeon Display Connectors
[    2.189219] [drm] Connector 0:
[    2.189220] [drm]   HDMI-A-1
[    2.189221] [drm]   HPD4
[    2.189222] [drm]   DDC: 0x6450 0x6450 0x6454 0x6454 0x6458 0x6458 0x645c 0x645c
[    2.189224] [drm]   Encoders:
[    2.189225] [drm]     DFP1: INTERNAL_UNIPHY2
[    2.189226] [drm] Connector 1:
[    2.189227] [drm]   DVI-I-1
[    2.189227] [drm]   HPD1
[    2.189228] [drm]   DDC: 0x6460 0x6460 0x6464 0x6464 0x6468 0x6468 0x646c 0x646c
[    2.189230] [drm]   Encoders:
[    2.189231] [drm]     DFP2: INTERNAL_UNIPHY
[    2.189232] [drm] Connector 2:
[    2.189232] [drm]   VGA-1
[    2.189233] [drm]   DDC: 0x6430 0x6430 0x6434 0x6434 0x6438 0x6438 0x643c 0x643c
[    2.189235] [drm]   Encoders:
[    2.189236] [drm]     CRT1: INTERNAL_KLDSCP_DAC1
[    2.274371] [drm] fb mappable at 0xD0363000
[    2.274373] [drm] vram apper at 0xD0000000
[    2.274374] [drm] size 8294400
[    2.274375] [drm] fb depth is 24
[    2.274376] [drm]    pitch is 7680
[    2.274429] fbcon: radeondrmfb (fb0) is primary device
[    2.274434] radeon 0000:01:00.0: [drm] fb0: radeondrmfb frame buffer device
[    2.279816] [drm] Initialized radeon 2.50.0 20080528 for 0000:01:00.0 on minor 0
[   15.309476] systemd[1]: Condition check resulted in Load Kernel Module drm being skipped.

Still having the same problem

[damentz]

Ok, one other thought then. Maybe the games you're playing have specific yield code? Try with:

echo 1 | sudo tee /proc/sys/kernel/yield_type and echo 2 | sudo tee /proc/sys/kernel/yield_type

Setting it to 1 yields the application to a higher priority task, while 2 always yields. Liquorix defaults to 0, which makes applications spin faster, which may be an issue on your system. Specifically, you have a 2008 CPU running games in 2021. Not only is it from 2008 and shouldn't be able to play most current games, but it's only a dual core, two thread CPU: https://www.cpu-world.com/CPUs/Core_2/Intel-Core%202%20Duo%20E7400%20AT80571PH0723M%20-%20AT80571PH0723ML%20(BX80571E7400).html

[Blackrucat]

That didn't work, I don't think it's about Yield, I'm still having problems, older versions of Liquorix it was playing normally. I am having problems mostly with Wine and Proton games, they're crashing or lagging, audio it's flickering a lot in those games. Games, like Brawhalla(which is a very low spec game), VrChat, and many others.

[damentz]

Maybe you're hitting a regression caused by having both futex2 and futex-multiple-wait in Zen? What happens to your performance if you add PROTON_NO_FSYNC=1 to your invocation of steam? Also while you're add it, you should see the fsync: up and running. if you don't disable fsync.

[Blackrucat]

I already do this to most of my WINE games.

[damentz]

Ok, is there a way you can test running a 5.11 kernel that's not Liquorix? You may be dealing with an upstream bug.

For Ubuntu you can pull the latest of 5.11.10 here: https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.11.10/

[Blackrucat]

I tried 5.11.9 generic and worked well.

I am not sure how to install from the link you sent me, so I searched, and I found how to install 5.11.9 and worked well, games are normal now.

I was having problems with games on Liquorix 5.11.9, so I think I can use 5.11.9 to compare those two kernels.

Games that I usually test(testing them on Generic 5.11.9): Brawlhalla: It's not really fast and the input is very delayed(I press some key during game, and it happens later), and it's hitting it's the best frame rate (60+fps) VrChat: Nice, playable! It's a bit slower than Windows, but that's Wine problem, it's not about the kernel.

[damentz]

Ok, thanks for trying mainline 5.11, that helps put things into context.

I reverted two patches that are recent to the 5.11/zen-sauce branch that might be the culprit. Can you try the next kernel version? (package: 5.11-22 / kernel: 5.11.0-10.2)

[Blackrucat]

(VR CHAT) Testing 5.11.22 and it's working! Thank you! I am just an average linux desktop user, and it's great to see Devs working with people that doesn't have much knowledge, and making the kernel better, this kernel is important for me, because I don't have money to have a good pc, and this gives me more FPS, thank you so much! DEVs like you make a difference in our lives. <3

[damentz]

Glad it's working well for you now! This also revealed that RCU configuration and tuning makes a big difference with gaming performance and framerate consistency, so I'll take a look at how all the major distributions and custom kernels configure RCU and figure out a better default.

[kerneltoast]

@damentz I just spent a few hours reviewing those two commits and I really can't see any issues with them.

@Blackrucat Can you reinstall the buggy kernel and, while running a laggy Steam game, run:

sudo timeout --foreground 30 perf top -d 1 -U --stdio > perf-top.output && sudo killall -s 9 perf

and upload the resulting perf-top.output file?

[Blackrucat]

Okay, I made huge mistake, I thought I booted on Liquorix, but I was booting on Generic(which is not lagging), so it's still lagging. Then I will try sudo timeout --foreground 30 perf top -d 1 -U --stdio > perf-top.output && sudo killall -s 9 perf in kernel: 5.11.0-10.2 because it has the same problem as before, so that wasn't RCU I am sorry for my mistake

[Blackrucat]

sudo timeout --foreground 30 perf top -d 1 -U --stdio > perf-top.output && sudo killall -s 9 perf timeout: falha ao executar o comando “perf”: Arquivo ou diretório inexistente

(it's saying perf file or directory do not exist)

[damentz]

@Blackrucat can you install the kernel headers? The perf tool is bundled with them and a profile script is added that prefers the tools bundled in the headers over what's installed on your system.

[Blackrucat]

I tried to perf only and it seems like I have perf tried to install headers

then i tried to install the headers it's already installed trying to reinstall

PERF still not working

even rebootin still having the same problem with perf

[damentz]

I think that's the Ubuntu sudo configuration blocking access to environment variables of your current user. You can run it directly with /usr/src/linux-headers-5.11.0-10.2-liquorix-amd64/tools/perf/perf, or you can sudo su - to root and run source /etc/profile.d/liquorix-tools.sh to bring it into your environment before running perf.

The line that's blocking you from using perf directly from sudo is in /etc/sudoers with the contents: Defaults secure_path="...". If you comment that out you can run sudo on anything as a regular user.

[Blackrucat]

Comment like this? Perf command worked but doesn't display any information

I think it was because of the headers and faulty things like perf, but perf command didn't made any difference. The game is not lagging anymore, but game audio still lagging (flickering)

[damentz]

Right, the perf output is what @kerneltoast was asking for. Can you compress it and attach it to this issue? It should be perf-top.output in your home folder.

[Blackrucat]

perf-top.output


   PerfTop:       0 irqs/sec  kernel: 0.0%  exact:  0.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------


   PerfTop:       1 irqs/sec  kernel:100.0%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------


   PerfTop:       0 irqs/sec  kernel: 0.0%  exact:  0.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------


   PerfTop:       0 irqs/sec  kernel: 0.0%  exact:  0.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------


   PerfTop:   10183 irqs/sec  kernel:12.3%  exact: 12.3% lost: 0/0 drop: 21109/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     2.73%  [kernel]       [k] __schedule
     2.33%  [kernel]       [k] _raw_spin_lock
     2.24%  [kernel]       [k] __fdget
     2.09%  [kernel]       [k] entry_SYSCALL_64
     1.84%  [kernel]       [k] native_write_msr
     1.76%  [kernel]       [k] entry_SYSENTER_compat
     1.69%  [kernel]       [k] copy_user_generic_string
     1.68%  [kernel]       [k] timerqueue_add
     1.45%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.44%  [kernel]       [k] __fdget_pos
     1.44%  [kernel]       [k] sysret32_from_system_call
     1.21%  [kernel]       [k] syscall_return_via_sysret
     1.20%  [kernel]       [k] try_to_wake_up
     1.12%  [kernel]       [k] _raw_spin_lock_irqsave
     1.12%  [kernel]       [k] skiplist_insert
     1.04%  [kernel]       [k] sock_poll
     0.96%  [kernel]       [k] evergreen_cs_parse
     0.96%  [kernel]       [k] entry_SYSENTER_compat_after_hwframe
     0.96%  [kernel]       [k] delay_tsc
     0.95%  [kernel]       [k] security_file_permission

   PerfTop:    8776 irqs/sec  kernel:13.2%  exact: 13.2% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     2.48%  [kernel]       [k] __schedule
     2.11%  [kernel]       [k] _raw_spin_lock
     1.98%  [kernel]       [k] __fdget
     1.60%  [kernel]       [k] native_write_msr
     1.58%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.54%  [kernel]       [k] entry_SYSCALL_64
     1.53%  [kernel]       [k] copy_user_generic_string
     1.32%  [kernel]       [k] __fdget_pos
     1.30%  [kernel]       [k] entry_SYSENTER_compat
     1.22%  [kernel]       [k] sock_poll
     1.21%  [kernel]       [k] timerqueue_add
     1.09%  [kernel]       [k] do_sys_poll
     1.04%  [kernel]       [k] delay_tsc
     1.03%  [kernel]       [k] syscall_return_via_sysret
     0.96%  [kernel]       [k] sysret32_from_system_call
     0.89%  [kernel]       [k] copy_page_range
     0.81%  [kernel]       [k] clear_page_rep
     0.77%  [kernel]       [k] _raw_spin_lock_irqsave
     0.77%  [kernel]       [k] page_vma_mapped_walk
     0.76%  [kernel]       [k] try_to_wake_up

   PerfTop:    7599 irqs/sec  kernel:12.2%  exact: 12.2% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     2.24%  [kernel]       [k] __schedule
     1.86%  [kernel]       [k] __fdget
     1.85%  [kernel]       [k] _raw_spin_lock
     1.70%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.52%  [kernel]       [k] __fdget_pos
     1.45%  [kernel]       [k] copy_user_generic_string
     1.36%  [kernel]       [k] native_write_msr
     1.36%  [kernel]       [k] entry_SYSCALL_64
     1.24%  [kernel]       [k] timerqueue_add
     1.20%  [kernel]       [k] entry_SYSENTER_compat
     1.19%  [kernel]       [k] delay_tsc
     1.18%  [kernel]       [k] clear_page_rep
     1.17%  [kernel]       [k] sysret32_from_system_call
     1.14%  [kernel]       [k] syscall_return_via_sysret
     1.13%  [kernel]       [k] sock_poll
     0.98%  [kernel]       [k] do_sys_poll
     0.91%  [kernel]       [k] _raw_spin_lock_irqsave
     0.88%  [kernel]       [k] entry_SYSCALL_64_after_hwframe
     0.86%  [kernel]       [k] try_to_wake_up
     0.81%  [kernel]       [k] copy_page_range

   PerfTop:    8975 irqs/sec  kernel:16.9%  exact: 16.9% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     2.11%  [kernel]       [k] _raw_spin_lock
     2.03%  [kernel]       [k] __fdget
     2.03%  [kernel]       [k] delay_tsc
     2.02%  [kernel]       [k] __schedule
     1.61%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.55%  [kernel]       [k] entry_SYSCALL_64
     1.51%  [kernel]       [k] syscall_return_via_sysret
     1.44%  [kernel]       [k] __fdget_pos
     1.41%  [kernel]       [k] copy_user_generic_string
     1.24%  [kernel]       [k] native_write_msr
     1.09%  [kernel]       [k] clear_page_rep
     1.08%  [kernel]       [k] _raw_spin_lock_irqsave
     1.04%  [kernel]       [k] do_sys_poll
     1.00%  [kernel]       [k] sysret32_from_system_call
     1.00%  [kernel]       [k] timerqueue_add
     0.97%  [kernel]       [k] entry_SYSENTER_compat
     0.96%  [kernel]       [k] entry_SYSCALL_64_after_hwframe
     0.94%  [kernel]       [k] sock_poll
     0.76%  [kernel]       [k] copy_page_range
     0.72%  [kernel]       [k] try_to_wake_up

   PerfTop:    7863 irqs/sec  kernel:15.6%  exact: 15.6% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     2.54%  [kernel]       [k] delay_tsc
     2.08%  [kernel]       [k] __fdget
     2.07%  [kernel]       [k] __schedule
     2.02%  [kernel]       [k] _raw_spin_lock
     1.80%  [kernel]       [k] clear_page_rep
     1.64%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.64%  [kernel]       [k] syscall_return_via_sysret
     1.62%  [kernel]       [k] entry_SYSCALL_64
     1.37%  [kernel]       [k] copy_user_generic_string
     1.35%  [kernel]       [k] __fdget_pos
     1.34%  [kernel]       [k] native_write_msr
     1.10%  [kernel]       [k] _raw_spin_lock_irqsave
     0.99%  [kernel]       [k] do_sys_poll
     0.95%  [kernel]       [k] sock_poll
     0.92%  [kernel]       [k] entry_SYSENTER_compat
     0.87%  [kernel]       [k] sysret32_from_system_call
     0.85%  [kernel]       [k] entry_SYSCALL_64_after_hwframe
     0.81%  [kernel]       [k] timerqueue_add
     0.78%  [kernel]       [k] copy_page_range
     0.77%  [kernel]       [k] iowrite8

   PerfTop:    7706 irqs/sec  kernel:13.3%  exact: 13.3% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     2.26%  [kernel]       [k] _raw_spin_lock
     2.21%  [kernel]       [k] delay_tsc
     1.96%  [kernel]       [k] __fdget
     1.91%  [kernel]       [k] __schedule
     1.82%  [kernel]       [k] clear_page_rep
     1.68%  [kernel]       [k] syscall_return_via_sysret
     1.62%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.51%  [kernel]       [k] entry_SYSCALL_64
     1.41%  [kernel]       [k] copy_user_generic_string
     1.29%  [kernel]       [k] native_write_msr
     1.22%  [kernel]       [k] __fdget_pos
     1.17%  [kernel]       [k] _raw_spin_lock_irqsave
     1.12%  [kernel]       [k] entry_SYSENTER_compat
     1.05%  [kernel]       [k] do_sys_poll
     0.92%  [kernel]       [k] page_vma_mapped_walk
     0.92%  [kernel]       [k] sock_poll
     0.85%  [kernel]       [k] __switch_to_asm
     0.84%  [kernel]       [k] sysret32_from_system_call
     0.82%  [kernel]       [k] timerqueue_add
     0.80%  [kernel]       [k] copy_page_range

   PerfTop:    9266 irqs/sec  kernel:11.9%  exact: 11.9% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     2.30%  [kernel]       [k] _raw_spin_lock
     2.29%  [kernel]       [k] delay_tsc
     1.87%  [kernel]       [k] __schedule
     1.76%  [kernel]       [k] clear_page_rep
     1.75%  [kernel]       [k] __fdget
     1.58%  [kernel]       [k] syscall_return_via_sysret
     1.54%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.35%  [kernel]       [k] entry_SYSCALL_64
     1.29%  [kernel]       [k] copy_user_generic_string
     1.25%  [kernel]       [k] native_write_msr
     1.11%  [kernel]       [k] entry_SYSENTER_compat
     1.10%  [kernel]       [k] page_vma_mapped_walk
     1.10%  [kernel]       [k] _raw_spin_lock_irqsave
     1.05%  [kernel]       [k] __fdget_pos
     0.96%  [kernel]       [k] do_sys_poll
     0.87%  [kernel]       [k] sysret32_from_system_call
     0.86%  [kernel]       [k] sock_poll
     0.86%  [kernel]       [k] timerqueue_add
     0.84%  [kernel]       [k] copy_page_range
     0.80%  [kernel]       [k] __switch_to_asm

   PerfTop:    6981 irqs/sec  kernel:10.9%  exact: 10.9% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     2.43%  [kernel]       [k] _raw_spin_lock
     2.07%  [kernel]       [k] delay_tsc
     2.04%  [kernel]       [k] clear_page_rep
     1.85%  [kernel]       [k] __schedule
     1.66%  [kernel]       [k] __fdget
     1.49%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.45%  [kernel]       [k] syscall_return_via_sysret
     1.31%  [kernel]       [k] entry_SYSCALL_64
     1.31%  [kernel]       [k] page_vma_mapped_walk
     1.26%  [kernel]       [k] copy_user_generic_string
     1.17%  [kernel]       [k] native_write_msr
     1.12%  [kernel]       [k] entry_SYSENTER_compat
     1.05%  [kernel]       [k] __fdget_pos
     1.02%  [kernel]       [k] _raw_spin_lock_irqsave
     0.92%  [kernel]       [k] do_sys_poll
     0.87%  [kernel]       [k] sysret32_from_system_call
     0.87%  [kernel]       [k] copy_page_range
     0.87%  [kernel]       [k] sock_poll
     0.85%  [kernel]       [k] timerqueue_add
     0.79%  [kernel]       [k] entry_SYSCALL_64_after_hwframe

   PerfTop:    8859 irqs/sec  kernel:10.9%  exact: 10.9% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     2.32%  [kernel]       [k] _raw_spin_lock
     2.15%  [kernel]       [k] clear_page_rep
     2.08%  [kernel]       [k] delay_tsc
     1.92%  [kernel]       [k] __schedule
     1.62%  [kernel]       [k] __fdget
     1.49%  [kernel]       [k] page_vma_mapped_walk
     1.40%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.37%  [kernel]       [k] syscall_return_via_sysret
     1.24%  [kernel]       [k] copy_user_generic_string
     1.22%  [kernel]       [k] entry_SYSCALL_64
     1.13%  [kernel]       [k] native_write_msr
     1.09%  [kernel]       [k] __fdget_pos
     1.04%  [kernel]       [k] entry_SYSENTER_compat
     0.99%  [kernel]       [k] _raw_spin_lock_irqsave
     0.89%  [kernel]       [k] __switch_to_asm
     0.88%  [kernel]       [k] copy_page_range
     0.86%  [kernel]       [k] timerqueue_add
     0.86%  [kernel]       [k] do_sys_poll
     0.85%  [kernel]       [k] sysret32_from_system_call
     0.82%  [kernel]       [k] entry_SYSCALL_64_after_hwframe

   PerfTop:    8697 irqs/sec  kernel:20.2%  exact: 20.2% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     3.09%  [kernel]       [k] entry_SYSENTER_compat
     2.44%  [kernel]       [k] sysret32_from_system_call
     2.33%  [kernel]       [k] delay_tsc
     2.18%  [kernel]       [k] _raw_spin_lock
     2.05%  [kernel]       [k] clear_page_rep
     1.68%  [kernel]       [k] __schedule
     1.44%  [kernel]       [k] __fdget
     1.37%  [kernel]       [k] copy_user_generic_string
     1.37%  [kernel]       [k] page_vma_mapped_walk
     1.33%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.17%  [kernel]       [k] entry_SYSENTER_compat_after_hwframe
     1.17%  [kernel]       [k] syscall_return_via_sysret
     1.11%  [kernel]       [k] entry_SYSCALL_64
     1.02%  [kernel]       [k] copy_page_range
     0.98%  [kernel]       [k] copy_page
     0.94%  [kernel]       [k] native_write_msr
     0.94%  [kernel]       [k] _raw_spin_lock_irqsave
     0.93%  [kernel]       [k] __fdget_pos
     0.80%  [kernel]       [k] read_tsc
     0.79%  [kernel]       [k] __get_user_nocheck_4

   PerfTop:   10319 irqs/sec  kernel:16.8%  exact: 16.8% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     3.01%  [kernel]       [k] clear_page_rep
     2.63%  [kernel]       [k] entry_SYSENTER_compat
     2.16%  [kernel]       [k] delay_tsc
     2.11%  [kernel]       [k] sysret32_from_system_call
     2.04%  [kernel]       [k] _raw_spin_lock
     1.63%  [kernel]       [k] __schedule
     1.44%  [kernel]       [k] __fdget
     1.39%  [kernel]       [k] copy_user_generic_string
     1.36%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.18%  [kernel]       [k] page_vma_mapped_walk
     1.16%  [kernel]       [k] syscall_return_via_sysret
     1.09%  [kernel]       [k] entry_SYSCALL_64
     1.01%  [kernel]       [k] entry_SYSENTER_compat_after_hwframe
     0.97%  [kernel]       [k] _raw_spin_lock_irqsave
     0.96%  [kernel]       [k] scan_swap_map_slots
     0.94%  [kernel]       [k] native_write_msr
     0.93%  [kernel]       [k] copy_page_range
     0.92%  [kernel]       [k] __fdget_pos
     0.83%  [kernel]       [k] get_page_from_freelist
     0.80%  [kernel]       [k] copy_page

   PerfTop:    8297 irqs/sec  kernel:19.7%  exact: 19.7% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     2.60%  [kernel]       [k] delay_tsc
     2.56%  [kernel]       [k] clear_page_rep
     2.55%  [kernel]       [k] entry_SYSENTER_compat
     1.92%  [kernel]       [k] _raw_spin_lock
     1.89%  [kernel]       [k] sysret32_from_system_call
     1.83%  [kernel]       [k] __schedule
     1.70%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.47%  [kernel]       [k] __fdget
     1.39%  [kernel]       [k] copy_user_generic_string
     1.31%  [kernel]       [k] syscall_return_via_sysret
     1.28%  [kernel]       [k] entry_SYSCALL_64
     1.07%  [kernel]       [k] _raw_spin_lock_irqsave
     1.02%  [kernel]       [k] native_write_msr
     0.96%  [kernel]       [k] page_vma_mapped_walk
     0.94%  [kernel]       [k] __fdget_pos
     0.91%  [kernel]       [k] entry_SYSENTER_compat_after_hwframe
     0.90%  [kernel]       [k] try_to_wake_up
     0.81%  [kernel]       [k] iowrite8
     0.79%  [kernel]       [k] copy_page_range
     0.79%  [kernel]       [k] do_sys_poll

   PerfTop:    8085 irqs/sec  kernel: 9.9%  exact:  9.9% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     2.68%  [kernel]       [k] delay_tsc
     2.36%  [kernel]       [k] clear_page_rep
     2.36%  [kernel]       [k] entry_SYSENTER_compat
     1.90%  [kernel]       [k] _raw_spin_lock
     1.82%  [kernel]       [k] __fdget
     1.81%  [kernel]       [k] __schedule
     1.78%  [kernel]       [k] sysret32_from_system_call
     1.74%  [kernel]       [k] entry_SYSCALL_64
     1.71%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.56%  [kernel]       [k] syscall_return_via_sysret
     1.42%  [kernel]       [k] copy_user_generic_string
     1.03%  [kernel]       [k] native_write_msr
     1.03%  [kernel]       [k] _raw_spin_lock_irqsave
     0.94%  [kernel]       [k] do_sys_poll
     0.93%  [kernel]       [k] __fdget_pos
     0.89%  [kernel]       [k] iowrite8
     0.88%  [kernel]       [k] try_to_wake_up
     0.87%  [kernel]       [k] page_vma_mapped_walk
     0.82%  [kernel]       [k] entry_SYSENTER_compat_after_hwframe
     0.81%  [kernel]       [k] sock_poll

   PerfTop:    8053 irqs/sec  kernel:14.6%  exact: 14.6% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     2.72%  [kernel]       [k] delay_tsc
     2.31%  [kernel]       [k] entry_SYSENTER_compat
     2.23%  [kernel]       [k] clear_page_rep
     2.01%  [kernel]       [k] __fdget
     1.87%  [kernel]       [k] native_queued_spin_lock_slowpath
     1.86%  [kernel]       [k] _raw_spin_lock
     1.82%  [kernel]       [k] __schedule
     1.73%  [kernel]       [k] entry_SYSCALL_64
     1.71%  [kernel]       [k] sysret32_from_system_call
     1.61%  [kernel]       [k] syscall_return_via_sysret
     1.46%  [kernel]       [k] copy_user_generic_string
     1.16%  [kernel]       [k] native_write_msr
     1.10%  [kernel]       [k] _raw_spin_lock_irqsave
     0.98%  [kernel]       [k] do_sys_poll
     0.92%  [kernel]       [k] iowrite8
     0.90%  [kernel]       [k] __fdget_pos
     0.90%  [kernel]       [k] try_to_wake_up
     0.84%  [kernel]       [k] __switch_to_asm
     0.82%  [kernel]       [k] sock_poll
     0.80%  [kernel]       [k] ioread8

   PerfTop:    7645 irqs/sec  kernel:17.9%  exact: 17.9% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     3.06%  [kernel]       [k] delay_tsc
     2.19%  [kernel]       [k] entry_SYSENTER_compat
     2.13%  [kernel]       [k] native_queued_spin_lock_slowpath
     2.13%  [kernel]       [k] __fdget
     2.12%  [kernel]       [k] clear_page_rep
     1.98%  [kernel]       [k] __schedule
     1.90%  [kernel]       [k] _raw_spin_lock
     1.66%  [kernel]       [k] sysret32_from_system_call
     1.63%  [kernel]       [k] entry_SYSCALL_64
     1.57%  [kernel]       [k] syscall_return_via_sysret
     1.39%  [kernel]       [k] copy_user_generic_string
     1.23%  [kernel]       [k] native_write_msr
     1.17%  [kernel]       [k] _raw_spin_lock_irqsave
     1.00%  [kernel]       [k] iowrite8
     1.00%  [kernel]       [k] do_sys_poll
     0.98%  [kernel]       [k] try_to_wake_up
     0.92%  [kernel]       [k] ioread8
     0.89%  [kernel]       [k] __fdget_pos
     0.81%  [kernel]       [k] __switch_to_asm
     0.78%  [kernel]       [k] entry_SYSCALL_64_after_hwframe

   PerfTop:    9132 irqs/sec  kernel:15.3%  exact: 15.3% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     3.09%  [kernel]       [k] delay_tsc
     2.37%  [kernel]       [k] native_queued_spin_lock_slowpath
     2.15%  [kernel]       [k] __schedule
     2.13%  [kernel]       [k] __fdget
     2.09%  [kernel]       [k] entry_SYSENTER_compat
     1.91%  [kernel]       [k] clear_page_rep
     1.86%  [kernel]       [k] _raw_spin_lock
     1.63%  [kernel]       [k] syscall_return_via_sysret
     1.59%  [kernel]       [k] sysret32_from_system_call
     1.57%  [kernel]       [k] entry_SYSCALL_64
     1.44%  [kernel]       [k] copy_user_generic_string
     1.27%  [kernel]       [k] native_write_msr
     1.21%  [kernel]       [k] _raw_spin_lock_irqsave
     1.09%  [kernel]       [k] try_to_wake_up
     1.05%  [kernel]       [k] iowrite8
     1.01%  [kernel]       [k] do_sys_poll
     0.95%  [kernel]       [k] ioread8
     0.91%  [kernel]       [k] __fdget_pos
     0.85%  [kernel]       [k] __switch_to_asm
     0.79%  [kernel]       [k] entry_SYSCALL_64_after_hwframe

   PerfTop:    8307 irqs/sec  kernel:19.5%  exact: 19.5% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     3.14%  [kernel]       [k] delay_tsc
     2.62%  [kernel]       [k] native_queued_spin_lock_slowpath
     2.26%  [kernel]       [k] __schedule
     2.05%  [kernel]       [k] __fdget
     2.01%  [kernel]       [k] entry_SYSENTER_compat
     2.00%  [kernel]       [k] clear_page_rep
     1.73%  [kernel]       [k] _raw_spin_lock
     1.59%  [kernel]       [k] copy_user_generic_string
     1.59%  [kernel]       [k] syscall_return_via_sysret
     1.55%  [kernel]       [k] entry_SYSCALL_64
     1.49%  [kernel]       [k] sysret32_from_system_call
     1.35%  [kernel]       [k] native_write_msr
     1.24%  [kernel]       [k] _raw_spin_lock_irqsave
     1.20%  [kernel]       [k] try_to_wake_up
     1.07%  [kernel]       [k] do_sys_poll
     1.06%  [kernel]       [k] iowrite8
     0.97%  [kernel]       [k] ioread8
     0.97%  [kernel]       [k] __fdget_pos
     0.84%  [kernel]       [k] timerqueue_add
     0.83%  [kernel]       [k] switch_fpu_return

   PerfTop:    8969 irqs/sec  kernel:19.5%  exact: 19.5% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     3.50%  [kernel]       [k] delay_tsc
     2.92%  [kernel]       [k] native_queued_spin_lock_slowpath
     2.27%  [kernel]       [k] __schedule
     1.99%  [kernel]       [k] __fdget
     1.96%  [kernel]       [k] entry_SYSENTER_compat
     1.72%  [kernel]       [k] clear_page_rep
     1.68%  [kernel]       [k] entry_SYSCALL_64
     1.66%  [kernel]       [k] _raw_spin_lock
     1.58%  [kernel]       [k] copy_user_generic_string
     1.53%  [kernel]       [k] syscall_return_via_sysret
     1.45%  [kernel]       [k] sysret32_from_system_call
     1.34%  [kernel]       [k] try_to_wake_up
     1.26%  [kernel]       [k] native_write_msr
     1.20%  [kernel]       [k] _raw_spin_lock_irqsave
     1.14%  [kernel]       [k] iowrite8
     1.05%  [kernel]       [k] __fdget_pos
     1.01%  [kernel]       [k] ioread8
     0.94%  [kernel]       [k] do_sys_poll
     0.90%  [kernel]       [k] entry_SYSCALL_64_after_hwframe
     0.88%  [kernel]       [k] timerqueue_add

   PerfTop:    6426 irqs/sec  kernel:23.2%  exact: 23.2% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     3.76%  [kernel]       [k] delay_tsc
     2.97%  [kernel]       [k] native_queued_spin_lock_slowpath
     2.36%  [kernel]       [k] __schedule
     1.89%  [kernel]       [k] __fdget
     1.86%  [kernel]       [k] entry_SYSENTER_compat
     1.71%  [kernel]       [k] clear_page_rep
     1.69%  [kernel]       [k] entry_SYSCALL_64
     1.64%  [kernel]       [k] _raw_spin_lock
     1.59%  [kernel]       [k] copy_user_generic_string
     1.55%  [kernel]       [k] syscall_return_via_sysret
     1.46%  [kernel]       [k] try_to_wake_up
     1.35%  [kernel]       [k] sysret32_from_system_call
     1.30%  [kernel]       [k] native_write_msr
     1.24%  [kernel]       [k] iowrite8
     1.24%  [kernel]       [k] _raw_spin_lock_irqsave
     1.05%  [kernel]       [k] __fdget_pos
     1.02%  [kernel]       [k] ioread8
     0.94%  [kernel]       [k] entry_SYSCALL_64_after_hwframe
     0.92%  [kernel]       [k] timerqueue_add
     0.85%  [kernel]       [k] do_sys_poll

   PerfTop:    8481 irqs/sec  kernel:19.5%  exact: 19.5% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     3.90%  [kernel]       [k] delay_tsc
     3.18%  [kernel]       [k] native_queued_spin_lock_slowpath
     2.33%  [kernel]       [k] __schedule
     1.91%  [kernel]       [k] __fdget
     1.76%  [kernel]       [k] entry_SYSENTER_compat
     1.72%  [kernel]       [k] entry_SYSCALL_64
     1.70%  [kernel]       [k] _raw_spin_lock
     1.61%  [kernel]       [k] copy_user_generic_string
     1.60%  [kernel]       [k] try_to_wake_up
     1.56%  [kernel]       [k] syscall_return_via_sysret
     1.52%  [kernel]       [k] clear_page_rep
     1.38%  [kernel]       [k] iowrite8
     1.30%  [kernel]       [k] sysret32_from_system_call
     1.30%  [kernel]       [k] native_write_msr
     1.25%  [kernel]       [k] _raw_spin_lock_irqsave
     1.08%  [kernel]       [k] ioread8
     1.03%  [kernel]       [k] __fdget_pos
     0.96%  [kernel]       [k] entry_SYSCALL_64_after_hwframe
     0.89%  [kernel]       [k] timerqueue_add
     0.88%  [kernel]       [k] do_sys_poll

   PerfTop:    8029 irqs/sec  kernel:16.9%  exact: 16.9% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     3.95%  [kernel]       [k] delay_tsc
     3.34%  [kernel]       [k] native_queued_spin_lock_slowpath
     2.34%  [kernel]       [k] __schedule
     1.94%  [kernel]       [k] __fdget
     1.74%  [kernel]       [k] entry_SYSCALL_64
     1.69%  [kernel]       [k] _raw_spin_lock
     1.69%  [kernel]       [k] entry_SYSENTER_compat
     1.64%  [kernel]       [k] try_to_wake_up
     1.64%  [kernel]       [k] copy_user_generic_string
     1.57%  [kernel]       [k] syscall_return_via_sysret
     1.38%  [kernel]       [k] iowrite8
     1.37%  [kernel]       [k] clear_page_rep
     1.34%  [kernel]       [k] native_write_msr
     1.32%  [kernel]       [k] sysret32_from_system_call
     1.25%  [kernel]       [k] _raw_spin_lock_irqsave
     1.11%  [kernel]       [k] ioread8
     1.04%  [kernel]       [k] __fdget_pos
     0.98%  [kernel]       [k] entry_SYSCALL_64_after_hwframe
     0.88%  [kernel]       [k] timerqueue_add
     0.88%  [kernel]       [k] __switch_to_asm

   PerfTop:    9221 irqs/sec  kernel:16.6%  exact: 16.6% lost: 0/0 drop: 0/21109 [4000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

     3.98%  [kernel]       [k] delay_tsc
     3.30%  [kernel]       [k] native_queued_spin_lock_slowpath
     2.36%  [kernel]       [k] __schedule
     2.02%  [kernel]       [k] __fdget
     1.81%  [kernel]       [k] entry_SYSCALL_64
     1.73%  [kernel]       [k] _raw_spin_lock
     1.70%  [kernel]       [k] syscall_return_via_sysret
     1.66%  [kernel]       [k] try_to_wake_up
     1.64%  [kernel]       [k] entry_SYSENTER_compat
     1.64%  [kernel]       [k] copy_user_generic_string
     1.34%  [kernel]       [k] native_write_msr
     1.30%  [kernel]       [k] iowrite8
     1.29%  [kernel]       [k] sysret32_from_system_call
     1.28%  [kernel]       [k] _raw_spin_lock_irqsave
     1.27%  [kernel]       [k] clear_page_rep
     1.03%  [kernel]       [k] ioread8
     0.97%  [kernel]       [k] __fdget_pos
     0.94%  [kernel]       [k] entry_SYSCALL_64_after_hwframe
     0.89%  [kernel]       [k] do_sys_poll
     0.88%  [kernel]       [k] timerqueue_add

[kerneltoast]

Hmm, the perf suggestion was in response to the RCU patches causing lags, possibly by burning CPU time. That perf output looks mostly fine, aside from some exceptional time spent in delay_tsc (which is a busy delay). It's probably not the cause for your lags, but it'd be nice to find out what's spending a lot of time busy looping.

Run this and upload perf-top.output again afterwards:

sudo timeout --foreground 30 perf top -d 1 -U --stdio -g --symbol delay_tsc > perf-top.output && sudo killall -s 9 perf

[kerneltoast]

@Blackrucat Actually, we should compare the perf output you provided with a kernel that doesn't have the lag issue. Install any kernel that doesn't have the lag (like vanilla 5.11 or something) and run this again:

sudo timeout --foreground 30 perf top -d 1 -U --stdio > perf-top.output && sudo killall -s 9 perf

[damentz]

Reopening for now.

[Blackrucat]

(5.11.9-051109-generic) Now I am having the same PERF problem with generic kernel :| Perf file doesn't exist. . . something like this

[damentz]

Sigh, that's right, Ubuntu requires a tools package and they only do it for their own kernels packaged part of the distribution. The mainline PPA packages don't have a tools package. It's actually ironic that their own package architecture for the kernel prevents you from using tools on their own mainline testing packages.

Aside from that, can you run perf from the Pop_OS kernel? That should have a tools package. It is much older (5.8 versus 5.11), but since you don't have issues then it should show us the difference we're looking for.

Also, check @kerneltoast's request in https://github.com/damentz/liquorix-package/issues/56#issuecomment-809580111. That'll get us more details into what's happening behind delay_tsc.

[Blackrucat]

I tried the 5.8 generic the default from pop!_os! and I am having the same problem with perf.

[damentz]

@Blackrucat try installing linux-tools-generic, that should provide perf for your distro kernel.

[Blackrucat]

I don't know why, my WINE games aren't opening anymore, in every kernel. It already happened before, they open and close immediately, that's not the first time it's happening, and I don't know if it has something to do with new WINE version or today 'oibaf ppa' drivers update. I think it might take a time for us to solve this problem I get back when I can, Linux is not helping me.

[damentz]

@Blackrucat are you able to continue trying the things @kerneltoast asked?

[damentz]

Let me know if you want to continue troubleshooting, but otherwise I'll close this issue.

[Blackrucat]

Yes I want, but I am to occupied studying, you can open it later. Sorry for being late to answer you, anyways you can let it close if you want and open again when I get some time to do this.