rpi-4.19-rt

[antonellocaroli]

there's going to be a 4.19 real time version?

[pelwell]

I hope so. We are currently relying on a volunteer - Tiejun Chen - to maintain the branch, and other commitments have meant that he hasn't had time recently.

[paul-1]

The RT patches apply cleanly to the 4.19.y branch. Been running pretty solid.

[vanfanel]

@paul-1 : I am trying to apply the RT patch to 4.19.y branch, too. Looking at the contents of the patch TAR in , I see there is a list of patches available:

0009-kthread-convert-worker-lock-to-raw-spinlock.patch
0012-arm-Convert-arm-boot_lock-to-raw.patch
0023-EXP-rcu-Revert-expedited-GP-parallelization-cleverne.patch
0028-sched-migrate_disable-Add-export_symbol_gpl-for-__mi.patch
0032-signal-Revert-ptrace-preempt-magic.patch
0036-rt-Provide-PREEMPT_RT_BASE-config-switch.patch
0052-x86-Use-generic-rwsem_spinlocks-on-rt.patch
0059-preempt-Provide-preempt_-_-no-rt-variants.patch
0061-rt-Add-local-irq-locks.patch
0066-buffer_head-Replace-bh_uptodate_lock-for-rt.patch
0067-fs-jbd-jbd2-Make-state-lock-and-journal-head-lock-rt.patch
0070-genirq-Disable-irqpoll-on-rt.patch
0076-mm-page_alloc-rt-friendly-per-cpu-pages.patch
0077-mm-swap-Convert-to-percpu-locked.patch
0098-time-hrtimer-avoid-schedule_work-with-interrupts-dis.patch
0099-hrtimer-consolidate-hrtimer_init-hrtimer_init_sleepe.patch
0100-hrtimers-Prepare-full-preemption.patch
0101-hrtimer-by-timers-by-default-into-the-softirq-contex.patch
0102-sched-fair-Make-the-hrtimers-non-hard-again.patch
0103-hrtimer-Move-schedule_work-call-to-helper-thread.patch
0104-hrtimer-move-state-change-before-hrtimer_cancel-in-d.patch
0105-posix-timers-Thread-posix-cpu-timers-on-rt.patch
0115-rt-Increase-decrease-the-nr-of-migratory-tasks-when-.patch
0128-rtmutex-trylock-is-okay-on-RT.patch
0130-rtmutex-Handle-the-various-new-futex-race-conditions.patch
0135-locking-locktorture-Do-NOT-include-rwlock.h-directly.patch
0136-rtmutex-Add-rtmutex_lock_killable.patch
0137-rtmutex-Make-lock_killable-work.patch
0139-rtmutex-Avoid-include-hell.patch
0141-rtmutex-Provide-rt_mutex_slowlock_locked.patch
0142-rtmutex-export-lockdep-less-version-of-rt_mutex-s-lo.patch
0143-rtmutex-add-sleeping-lock-implementation.patch
0144-rtmutex-add-mutex-implementation-based-on-rtmutex.patch
0145-rtmutex-add-rwsem-implementation-based-on-rtmutex.patch
0146-rtmutex-add-rwlock-implementation-based-on-rtmutex.patch
0147-rtmutex-rwlock-preserve-state-like-a-sleeping-lock.patch
0148-rtmutex-wire-up-RT-s-locking.patch
0149-rtmutex-add-ww_mutex-addon-for-mutex-rt.patch
0151-locking-rt-mutex-fix-deadlock-in-device-mapper-block.patch
0152-locking-rt-mutex-Flush-block-plug-on-__down_read.patch
0153-locking-rtmutex-re-init-the-wait_lock-in-rt_mutex_in.patch
0155-rtmutex-annotate-sleeping-lock-context.patch
0168-rt-Improve-the-serial-console-PASS_LIMIT.patch
0183-rt-Introduce-cpu_chill.patch
0184-hrtimer-Don-t-lose-state-in-cpu_chill.patch
0185-hrtimer-cpu_chill-save-task-state-in-saved_state.patch
0196-seqlock-Prevent-rt-starvation.patch
0197-sunrpc-Make-svc_xprt_do_enqueue-use-get_cpu_light.patch
0207-printk-Make-rt-aware.patch
0214-kgdb-serial-Short-term-workaround.patch
0216-mm-rt-kmap_atomic-scheduling.patch
0219-arm-Enable-highmem-for-rt.patch
0227-x86-stackprotector-Avoid-random-pool-on-rt.patch
0228-random-Make-it-work-on-rt.patch
0240-sched-Add-support-for-lazy-preemption.patch
0242-x86-Support-for-lazy-preemption.patch
0245-arm-Add-support-for-lazy-preemption.patch
0246-powerpc-Add-support-for-lazy-preemption.patch
0247-arch-arm64-Add-lazy-preempt-support.patch
0249-drivers-block-zram-Replace-bit-spinlocks-with-rtmute.patch
0254-drm-radeon-i915-Use-preempt_disable-enable_rt-where-.patch
0259-cpuset-Convert-callback_lock-to-raw_spinlock_t.patch
0262-signals-Allow-rt-tasks-to-cache-one-sigqueue-struct.patch
0264-Linux-4.19.37-rt19-REBASE.patch

What set of patches do you apply to get a stable RT kernel on the Pi?

[antonellocaroli]

I applied this without any problems:

https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/4.19/patch-4.19.37-rt19.patch.gz

[vanfanel]

@antonellocaroli : Great! What other options, appart from GENERAL SETUP->PREEMPTION MODEL can you recommend me to change? In KERNEL FEATURES->TIMER FREQUENCY, I already have 1000Hz (default in 4.14.y and 4.19.y branches anyway).

[antonellocaroli]

@vanfanel you just need them... but take a look at this to find out more: https://rt.wiki.kernel.org/index.php/Frequently_Asked_Questions

[vanfanel]

@antonellocaroli : Thanks! I read that document time ago. What I usually do to set realtime priority on a process is: 1) Set the rlimits system-wide accordingly so normal users can set realtime priority. 2) Execute the process with chrt -f 80 <executable name> If you have any other observation regarding RT on GNU/Linux, please share it.

[guysoft]

Following, so I can get RealtimePi to point to it when ready

[NikiSchlifke]

+1

[guysoft]

Just realised no one actually mentioned @TiejunChina . So no wonder this has been hanging for 25 days.

[pelwell]

Tiejun is well aware of the situation. Pinging him (which I intentionally avoided) won't help.

[guysoft]

@pelwell Understood, it did not happen here and I could not guess at your private communication. Apologies.

I recommend notifying that next time, since I was starting to speculate on making my own PR branch.

[pelwell]

I was starting to speculate on making my own PR branch.

I wouldn't have a problem with that, if you can follow the same format as Tiejun did. The standard rpi- branches use merge commits to pull in upstream patches so they keep their commit IDs, and cherry-ick/rebase downstream commits to keep the commit history clear - you end up with two parallel streams of commits, "us" and "them".

As I wrote to Tiejun at the time:

"One of our internal rules is that, apart from back-porting patches to older kernels, we never change the upstream commits. This means that our development looks like this:

    L-L-L-L-L-L-L---L-L-L-L-L-L-L     = Upstream Linux
                 \       \       \
                  P-P-P-P-M-P-P---M-P = Downstream rpi- branch
                      ^   ^       ^
                      A   B       C

where L is an upstream Linux commit, P is a downstream Pi commit, and M is a merge commit. A represents the point where we stop rebasing our commits against the new kernel version, which corresponds to the point where it becomes our mainline branch. B and C are minor kernel version updates.

"Perhaps you could structure the -rt branch like this:

    L-L-L-L-L-L-L----L-L-L-L-L-L-L     = Upstream Linux
                \         \       \
                 R-R-R-R-M-M--M----M-M = rpi-rt branch
                        /    /     /
                 P-P-P-P--P-P-----P    = Downstream rpi- branch

"I hope you can understand my diagram. The important point is that the L and P commits in this diagram are identical to those in the previous one, with the same commit hash. The nice thing about this scheme is that by not rebasing your existing commits you only need to work on and think about (and fix) new commits."

If you can follow that pattern and prepare a branch in your repo then I'll be happy to bless it and call it rpi-4.19-rt. If, in doing so, you end up creating any tools to simplify the task then let me know - should it become easy enough then eventually the task may be brought in-house.

[paul-1]

Merging Upstream Linux commits into the RT branch is fun. I've never kept a copy of the rt-kernel git.

When maintaining for myself, when there is a new -rtxx version, I start from a clean rpi branch, and apply the rt patch set, then cherry pick any rpi-rt commits. I know that some don't like constant rebasing, but in my mind keeps the rt branch only a few commits different than the normal rpi branch.

[TheLongAndOnly]

@paul-1 As I understood you managed to get the RT patches to work for 4.19. I tried myself but ran into problems and hope you might have some hints.

As patching 4.19.y HEAD did not work, I pulled the matching commit for 4.19.37. These are the steps I took:

• git clone https://github.com/raspberrypi/linux.git --single-branch --branch rpi-4.19.y
• git checkout 19bb613acb9ad8e57593cad5118acaee117cc303 (this is the commit changing to version 4.19.37)
• make bcm2709_defconfig (had to pull it from head, as it is not part of the commit) As the debos are not build, I updated to the correct commit of rpi-update
• sudo rpi-update 6cf2788e74b497c109865d052cd6fa68d094cf38

Unfortunately after installation of the new kernel I do not get past the rainbow screen. Changing kernel7.img back to the original one makes the system boot.

[guysoft]

I am not an expert, but looking over the tty serial output might give you more information to what is failing at the rainbow screen.

[paul-1]

RPI is rebasing the 4.19.y tree, you cannot go back like that right now, unless you go and then pick all of the rpi downstream commits. did you try to apply the 4.19.37-rt19 against the current branch? Otherwise it's best to wait until the next rt patch comes out.

[TheLongAndOnly]

Ok, I didn't realize this, but it explains why the configs are missing. Applying 4.19.37-rt19 to the current branch does not work as there have been changes to mainline which makes the build fail.

[paul-1]

4.19.50-rt22 is out, it applies cleanly against the current rpi branch.

[pelwell]

Thanks for the tip.

[guysoft]

@paul-1 Isn't the dwc_otg fix patch also needed? https://www.osadl.org/Single-View.111+M5c03315dc57.0.html I have it in source control here: https://github.com/guysoft/RealtimePi/blob/8b375fd1848276fa22c5c24a490d9696cd4a9e85/src/modules/realtimepi/filesystem/home/pi/patches/0001-usb-dwc_otg-fix-system-lockup-when-interrupts-are-th.patch

[paul-1]

I have not been applying that for a while, and definitely not on the 4.19 branch, I have not noticed a problem. But I've not done a ton of heavy USB testing.

Edit: You may be correct, I think with all the 4.19 changes, I made a mistake in my defconfig.

[pelwell]

Please take a look at https://github.com/pelwell/linux/tree/rpi-4.19.y.rt . It's a simple merge of the current rpi-4.19.y.rt with the RT commits on top. I've also copied across one of Tiejun's commits. Other than building it and booting it on a 3B+ I've not done any real testing, but if people think it's OK then I'm happy to push it as a semi-official, low-priority-supported branch off raspberrypi/linux.

[paul-1]

Should you change the defconfigs in that branch for
CONFIG_PREEMPT_RT_FULL=y

[pelwell]

Good point. I've updated the branch with updated defconfigs, but without the cherry-pick of Tiejun's patch (it was locking up for me with that included).

[pelwell]

I'm also using dtoverlay=dwc2.

[guysoft]

I'll try find the time and make a build or RealtimePi, so we can all be testing the same thing.

[paul-1]

I'm using the standard dwc_otg driver (With fiq & fsm enabled).......I've never had good luck with the dwc2.

I've confirmed that the patch "don't thread shared irqs" from Tiejun is needed if SPI is enabled. I also applied the patch from osadl.org, but there has been changes to the dwc_otg driver that still hangs the system. I believe this commit adds a spot that needs to be locked with the fiq_fsm_spin_lock_irqsave 3ea9af8bc4eaac0cca72d603ed081973bce91bcf after patching that part, kernel has been stable.....but doing more testing now.

[pelwell]

Thanks for the feedback. You could probably save some time and effort if you could submit a Pull Request with the necessary changes, otherwise I'll have another go when I can.

[paul-1]

I will once I do some validation.....I don't want to make mistakes that I made with earlier RT builds.

[pelwell]

So that everyone is up-to-speed, two things happened early today:

1. @paul-1 submitted a PR (thanks!) to my candidate RT branch: https://github.com/pelwell/linux/pull/1
2. @TiejunChina (welcome back! (-8) created an "official" rpi-4.19.y-rt branch.

1) works for me (without the dwc2 overlay), but only if I revert the "no threaded irq" patch. 2) also needs reverting for me, along with an additional small patch to use fiq_fsm_spin_lock_irqsave in dwc_otg_read_common_intr (this is in paul-1's patch set).

One possible difference between my setup and yours is that I use the UART console on ttyS0 (the mini-UART with the shared IRQ) - running with dtoverlay=pi3-disable-bt switches the console to ttyAMA0/UART0 and prevents the lockup. A partial reversion - not threading the SPI interrupt but leaving the UART interrupt - is also effective, but I'm not using SPI; more testing is needed.

[guysoft]

Got a nightly build out with the new kernel: Image: edit: nightly build here did not work, see newer build

Build script sources for the rpi-4.19-rt branch: https://github.com/guysoft/RealtimePi/tree/rpi-4.14.y-rt

[guysoft]

@pelwell Do you have the "no threaded irq" patch or link to commit, so I can revert it for the next nightly build?

[pelwell]

Of course - it's ca8809033b8f69f5999f3407f4f39667f07d7bee. To be clear, I'm not saying that we don't need some variant of that commit, but that it isn't correct as it stands.

[guysoft]

Ok, I am going to revert https://github.com/raspberrypi/linux/commit/ca8809033b8f69f5999f3407f4f39667f07d7bee and make another build.

BTW I just tested the nightly build without it. And it kernel panics on boot. This is what I seem to be getting:

and with kgdboc=ttyS0,115200 in cmdline.txt:

[guysoft]

Ah wait, https://github.com/raspberrypi/linux/commit/ca8809033b8f69f5999f3407f4f39667f07d7bee is already in the rpi-4.19-rt, so it was shipped in the image.

Ok, so basically we are stuck because of the kernel panic above. I am not a kernel developer, so if anyone wants to help out and explain to me what is going wrong, and how to fix it, it could help make the branch usable.

[paul-1]

I'm not a kernel developer either, but I can follow code, and compare when things break.....

The branch 4.19.y-rt in the raspberrypi/linux git won't work as is if you have FIQ enabled. If that is the branch you are using,

If SPI is disabled, then you can revert. https://github.com/raspberrypi/linux/commit/ca8809033b8f69f5999f3407f4f39667f07d7bee] If SPI is enabled, then you can not revert that commit, and can not use the serial console until things get figured out.

IF FIQ is enabled, then you also need to add https://github.com/pelwell/linux/pull/1/commits/02586ccd069c7f0e7be7be4610a1207295ca9bc5

[guysoft]

How would I disable FIQ? Is that a good idea at all? I am trying to have a distribution you can just flash and use this branch. I think most people use it with Jack2, so that would be my focus. Would FIQ effect USB devices?

[paul-1]

Add to command line dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0.

In the early single core days, the usb was nearly unusable without fiq....not sure how newer boards would run without it, let alone a RT kernel. But since it's still default for rpi to use fiq, I leave it enabled.

[guysoft]

@paul-1 Ok, adding dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 makes it hang in a different location now. Looks like it hangs around the LAN driver. but the kernel panic is the same.

and with kgdboc=ttyS0,115200 in cmdline.txt:

[guysoft]

So this kinda went quiet because of the RaspberryPi 4 release - any opinters what is going wrong now?

[TheLongAndOnly]

A while ago I managed to get 4.19.y to work with the older RT patch. I will try to look up the config parameters tomorrow. Latency has not been great, but it worked stable. Unfortunately my build PC died, so I have to get it from the SD.

[TheLongAndOnly]

This is the command line I use to start: dwc_otg.lpm_enable=1 console=serial0,115200 console=tty1 root=PARTUUID=46877fa5-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait splash plymouth.ignore-serial-consoles likely only the first entry really has an effect.

I used the rt20 patch on kernel 4.19.46 and had to modify one file to make it compile. I'm pretty sure I had the usb-dwc_otg-fix patched ontop.

[guysoft]

Ok, using @TheLongAndOnly 's make the system boot! But with no USB working. Tested on a RaspberryPi A+.

I had to change:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=c1dc39e5-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh

To:

dwc_otg.lpm_enable=1 console=serial0,115200 console=tty1 root=PARTUUID=c1dc39e5-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait splash plymouth.ignore-serial-consoles

Uname gives over ssh:

Linux realtimepi 4.19.50-rt22-v7 #2 SMP PREEMPT RT Thu Jun 20 07:50:16 BST 2019 armv7l GNU/Linux

I will update cmdline.txt and make a new build. However something seems unsable.

[guysoft]

Ok, I have a nightly that boots, using the branch. uname:

Linux realtimepi 4.19.50-rt22-v7 #2 SMP PREEMPT RT Sat Jun 29 10:17:23 BST 2019 armv7l GNU/Linux

What didnt' work:

• keyboard
• ethernet

what worked:

• boots
• wifi and ssh
• 3B+
• 3A+

Download at: http://unofficialpi.org/Distros/RealtimePi/nightly/2019-06-29_2019-06-20-realtimepi-buster-lite-0.4.0.zip md5: http://unofficialpi.org/Distros/RealtimePi/nightly/2019-06-29_2019-06-20-realtimepi-buster-lite-0.4.0.zip.md5

Binary: http://unofficialpi.org/Distros/RealtimePi/nightly/2019-06-29_realtimepi-kernel-4.19.50.tar.gz

Also upgraded to buster.

I think we need to add the usb-dwc_otg-fix for the keyboard to waork.

[pelwell]

We don't have a lot of time for diagnosis at the moment, but I'll happily merge a PR for any required changes.

[guysoft]

I am also kinda busy at the moment, but I can offer these builds of the branch for testing since I have the distro in place.

Anyone here planning to play with it?

[paul-1]

It's a bit out of my understanding to play with interrupts. What I have works for our application (piCorePlayer) as most of our users are not interested in using the serial line in a RT environment. That appears to be the big problem maker.

Now if I could just get my hands on my pi4 shipments, I could get to work.

[TiejunChina]

Ok, I have a nightly that boots, using the branch. uname:

Linux realtimepi 4.19.50-rt22-v7 #2 SMP PREEMPT RT Sat Jun 29 10:17:23 BST 2019 armv7l GNU/Linux

What didnt' work:

• keyboard
• ethernet

what worked:

• boots
• wifi and ssh
• 3B+
• 3A+

Download at: http://unofficialpi.org/Distros/RealtimePi/nightly/2019-06-29_2019-06-20-realtimepi-buster-lite-0.4.0.zip md5: http://unofficialpi.org/Distros/RealtimePi/nightly/2019-06-29_2019-06-20-realtimepi-buster-lite-0.4.0.zip.md5

Binary: http://unofficialpi.org/Distros/RealtimePi/nightly/2019-06-29_realtimepi-kernel-4.19.50.tar.gz

Also upgraded to buster.

I think we need to add the usb-dwc_otg-fix for the keyboard to waork.

@guysoft I'm got stuck in my regular work so sorry for this inconvenience. Did you try our rpi-4.14.-rt? Even we decided not to upgrade that but I recall 'keyboard' and 'ethernet' both worked on my 3b board.

[guysoft]

@TiejunChina Hey, good to see you back. Yes I have an RC for that: https://github.com/guysoft/RealtimePi/issues/16

We will need though eveltually to move to rpi-4.19-rt.