after installing kernel-lqx-mao other kernels fail to boot on FC39

[henning]

In the search for Kernels that help me get recording many (at least 24, real goal is around 60 from 3 devices) parallel audio channels via usb interfaces done, I found the liquorix kernel and audinux as a way to get it installed.

I worked by this description on Fedora 39: https://copr.fedorainfracloud.org/coprs/ycollet/audinux/

After installing kernel-lqx-mao, I realize that the audio recording performance is actually even worse, creating noise when using pipewire and/or creating xruns when using jack.

Now I dont write this to debug the performance. My problem now is, I want to go back to the standard kernels, but they dont boot anymore, this happens:

When selecting the normal fedora kernels(I have multiple installed - 6.6.6 6.6.9 etc), in the grub menu, they dont boot, but hang and reboot itself with the last message being seen is:

[FAILED] Failed to mount tmp.mount - Temporary Directory /tmp
[FAILED] Failed to start dbus-broker.service - D-Bus message bus

That stays a few seconds, then the system reboots itself.

I'm not new to linux, but have not much experience on Fedora. I see when using the liquorix kernel, /tmp is a tmpfs mount.

So I tried to find where that is defined in order to better understand why it works with lqx kernel, but not with others. But, it's not in fstab, and I cannot find any place in /etc or even /boot that looks like being responsible to create that mount / tmpfs in /tmp. I grepped and search around, but I dont find a solution.

I hope I dont miss seeing anything totally obvious and waste your time... it looks like a trivial problem but I dont get it.

[ycollet]

Thanks for the report. I have not tested the last liquorix kernel. Last I checked, it worked. I will try to reproduce the problem and fix it next week. In the mean time, you can try the standard fedora kernel. There are some real time patch inside now. You need to perform somes steps before that:

• change the priorities in /etc/security/limits.d/25-pw-rlimits.conf:

  @pipewire   - rtprio  95
  @pipewire   - nice    -19
  @pipewire   - memlock 4194304

• add yourself to the pipewire group. As a root user:

  $ usermde -a -G pipewire <YOUR_USER_NAME>

  The restart your computer and, before booting Fedora, add these options to the kernel you will boot:

  preempt=full threadirqs

  On my side, I will try to check the liquorix kernel.

[henning]

wait, i think thats a misunderstanding.

the liquorix kernel works for me. At least it boots, and things look normal, but does not provide the performance i hoped for.

so i want to use normal kernels again, but after having kernel-lqx-mao installed, all other kernels don't boot anymore.

the symptom i described is with booting back to a normal Fedora Kernel. I tried multiple versions, also reinstalling them.

but always the same issue.

my question is if you are aware of any changes the lqx kernel package or its dependencies do to the system that van cause these problems with the standard kernel as i described.

[ycollet]

Ah, OK, sorry. For me; there is not configuration changes when you install the liquorix kernel. The process is just installation of kernel + kernel module and launch of grub to add a line in the grub boot loader. The problem related to the tmpfs not mountable is strange. I will try to find some more informations on that and I'll be back to the discussion.

[ycollet]

So, I just tried the Liquorix kernel. Worked fine wrt audio performances. But when I rebooted on Standar Fedora kernel, a SE Linux fle system relabelling was triggered. This process took some time. But after the SE Linux relabelling, I was able to boot on the standard Fedora Kernel. So, there is a problem with the Liquorix kernel. I bet it need a patch applied on the standard kernel and related to SE linux. I will check next week.

[ycollet]

Something you can try:

• boot on Liquorix.
• reboot once on liquorix
• at boot time select standard fedora
• wait until relabelling is done.
• Once relabelling is done you are back on grub and now you can select standard fedora kernel

If you miss the grub menu, your linux will reboot on liquorix.

Once on standard fedora, don't forget to remove the liquorix kernel ...

[henning]

Thanks for your fast response - I only now found time to get to the computer to try these steps.

But it doesn't seem to help.

At step 4 - wait until relabelling is done - I was not sure what I can do that this relabeling is really happening, and I cannot "wait" for anything, because the system always reboots on its own after the error message from my first mesage here is show for a very short time, just 1s or so. I needed to take a foto to even be able to read the message properly.

But I tried anyway to boot the standard fedora kernel multiple times in a row. It's always the same behaviour.

I took photos of the messages again, and what I can see is that there is actually a message about filesystem relabeling.

but it's only /dev /dev/shm, /run, /sys/fs/cgrp - the /tmp that is appearing in the last error messge before the boot stops and reboots is not in this list.

Can i do something to enforce this?

Can these problems have something to do that my root filesystem is btrfs and it's encrypted? (for the sound stuff I use another partition that is not encrypted, so I assume/hope that's not the performance problem. I do it that way with other systems to)

I will also try your suggestions from above to see if they help to improve performance on lqx kernel, but i'd still want to find out how to boot other kernels again ;)

I found that there are more rt kernels in the audinux repo that I could try to see which of these options is best for my performance, but they have the same boot problem as the standard fedora kernels).

[henning]

What I also tried is disabling selinux to confirm that the problem lies somewhere in that area.

And yes, with selinux disabled, the standard Kernel boots properly.

I also thought maybe I can try to see if after booting that way twice i can check if it boots then when i re-enable SELinux, but that didn't work...

I also tried to do a "touch /tmp/.autorelabel" to enforce the relabeling of /tmp, but that had no effect either. But there is already a /.autorelabel file, so it should happen for everything.

[ycollet]

If you are able to reboot on standard kernel, can you try to use as a root user:

$ restorecon -p -r /

And another question: is your system partition full or not ?

[henning]

that doesn't change anything, after running this command and trying again to boot the standard Fedora kernel with selinux enabled as it's the default, it has the same error as described in the first message.

system partition is not full, many GB free space. Other partitions are also not full.

[ycollet]

Have you removed the liquorix kernel ?

[henning]

Yes I did remove it, tried everything again, but it didn't change anything.

Your explanation that it's just the Kernel and that should not have any effect when another kernel is booted sounds logical. So I installed it again after that as I also want to debug the performance issue further.

Slightly Off Topic, but as the website of audinux says to install the kernel-rt-mao which is a bit older - which of the three kernels from the audinux repository is the most recommended? I also tried the other two, but currently they have the same problem as the standard fedora Kernel - they only boot when SELinux is off.

[ycollet]

Since the recent releases of pipewire and the kernel, I am now using the Fedora kernel with 2 options at boot time : preempt=full and threadirqs and I replaced jack by pipewire jack. I am quite satisfied by the real time audio performances now. That was not the case at all when I started to use pipewire. I also stopped building the kernel-rt 5.15 because there were a. lot of problems with the audio drivers. I put a warning related to the liquorix kernel both on the website and on the COPR website. I will try to check again if I can find a solution to your problem. I think I will open a ticket concerning the SE Linux problem on the Liquorix git repository.

[ycollet]

I was checking the differences between Liquorix and standard kernel config options. Can you try to add the following option at boot time ?

lsm=lockdown,yama,integrity,selinux,bpf,landlock

This is the only difference I checked between these kernels.

[henning]

Yes! With these two changes the Fedora kernel actually shows the behaviour you described earlier - a selinux relabling happens that takes some while, then the system reboots automatically, and it boots into an apparently working state.

I can also switch back and forth between the normal and the lqx kernel, and it seems that the relabeling only took a few minutes once, right now i switched back from lqx to Fedora kernel and it booted just quickly.

I also tested if this situation is reproducable, by installing the audimux repository and the lqx kernel on another Fedora 39 testing machine.

But there the situation didn't happen. Even though the suggested changes to the kernel commandline do not exist there, the system can boot back and forth between Fedora kernel and lqx kernel.

So it's still a mystery what exactly caused this situation...

But I made enough reboots now :)

For me it looks solved now, but please let me know if you want to do anything further, or if I shall test anything else to find the root cause.

I will make some tests to see if your suggestions to the kernel command line get me further with performance/recording stability issues.

Thanks a lot for the fast help!

[henning]

One more observation - even though it doesn't make things more clear:

As I saw that one machine had no problem booting between lqx and fedora kernel also without the special commandline you proposed above, i also tried what happens tp the computer that got that problem before, if I remove the

lsm=lockdown,yama,integrity,selinux,bpf,landlock

addition again.

I observed then the machine with the error also had no problem anymore.

So, there was something that somehow prevented that necessary relabeling from happening, but once the relabeling could run with this added commandline parameters, they were not necessary anymore.

I have no idea why this is so... but in general I also dont fully understand how grub in Fedora works anyway, because I dont see the "usual" menuentries for kernels anymore in /boot/grub.conf and dont understand how commandlines and other parameters are handled at boot time.

Anyway, even if we didn't find the root cause how this all got triggered, it's done for me.

I will happily run any kind of test you want me to in case you want to further search for the root of this problem but close the issue for the time being...

[ycollet]

I repon the ticket and will closes it once I found a solution to boot liquorix without the SE Linux problem. Thanks for your patience and I am heppy to see that the problem is gone :)

[ycollet]

Fixed with the last update ... I close the ticket