PX4 process isolation on Linux boards

[blazczak]

The px4 process on Linux (px4_raspberrypi, emlid_navio2) would benefit from basic adjustments to the scheduler to increase task priority and to isolate the autopilot process from other non-critical processes, at the same time improving responsiveness for other ad-hoc processes and command-line. This can also translate to improved reliability and flight safety.

Navio2 is supported on Raspberry Pi boards starting with version 2, which means all of its deployments are multi-core. With the current state of tech it can be safely presumed that going forward every PX4 deployment on Linux will be on multi-core boards.

I suggest creating and certifying a script in PX4 that does the following before launching the px4 binary:

1. Sets px4 process priority to maximum using nice -19
2. Uses taskset to isolate the px4 process into a dedicated core with chosen non-zero index (kernel will use zero-index core for some its tasks)
3. Removes interrupts from the px4-only core

This is explained in more detail, with an example of a ready-to-use script, at http://www.hydrogen18.com/blog/howto-give-a-single-process-its-own-cpu-core-in-linux.html

The scripts and install/configuration steps should be included in the docs at https://docs.px4.io/master/en/flight_controller/raspberry_pi_navio2.html

I didn't check if Emlid was already doing this in their Raspberry Pi OS images; if not, they could certainly benefit.

[dagar]

If you run as root PX4 on Linux should already be the the highest priority in the system because it's using real time threads (SCHED_FIFO). It would be interesting to see if there are additional benefits to gain.

To casually compare you can take a look at perf counters (perf), in particular control latency to get a measure for end-to-end latency from gyro input to actuator outputs written.

[blazczak]

@dagar From the output you had asked for in #16006 it can be seen that regularly prioritized px4 running as root is at priority 20 (PR 20 NI 0)

Here's top with the above suggestions (PR 1 NI -19):

top - 04:49:10 up 1 min,  1 user,  load average: 7.93, 2.46, 0.87
Tasks: 143 total,   4 running, 139 sleeping,   0 stopped,   0 zombie
%Cpu(s):  9.7 us, 36.8 sy,  0.0 ni, 53.2 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
MiB Mem :    801.5 total,    344.8 free,    148.2 used,    308.5 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.    630.7 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND   
 1405 root       1 -19   70432   4060   3644 R  99.0   0.5   0:33.12 px4       
  150 root      20   0       0      0      0 R  19.9   0.0   0:06.26 spi0      
    1 root      20   0   33468   7772   5312 S  15.3   0.9   0:15.18 systemd   
  651 root      20   0       0      0      0 D  13.4   0.0   0:12.56 rcio_work+
 1110 message+  20   0    6392   2736   2368 S  11.7   0.3   0:03.88 dbus-daem+
 1275 ubuntu    20   0   51512  16412  14288 S   7.5   2.0   0:04.33 gst-launc+
 1122 root      20   0   12360   5212   4556 S   5.5   0.6   0:02.11 systemd-l+
  188 root      20   0       0      0      0 S   2.9   0.0   0:02.78 spi1      
 1785 ubuntu    20   0    8392   2388   2044 R   2.3   0.3   0:00.32 top       
   10 root      20   0       0      0      0 I   1.0   0.0   0:01.10 rcu_preem+
 1022 systemd+  20   0   21648   3064   2496 S   1.0   0.4   0:01.06 systemd-t+
  108 root       1 -19       0      0      0 S   0.7   0.0   0:00.46 vchiq-slo+
  116 root      20   0       0      0      0 I   0.7   0.0   0:00.09 kworker/u+
    9 root      20   0       0      0      0 S   0.3   0.0   0:00.80 ksoftirqd+
   99 root      20   0       0      0      0 I   0.3   0.0   0:00.29 kworker/1+
  100 root      20   0       0      0      0 I   0.3   0.0   0:00.19 kworker/3+
  132 root      20   0       0      0      0 R   0.3   0.0   0:00.29 kworker/2+

Additionally, I'm not only suggesting prioritization, but also process isolation through mapping into a dedicated core and preventing the process in that core from being interrupted.

I see a significant improvement in response with the changes I'm suggesting above.

[SalimTerryLi]

Maybe you want to have a look at htop. Only one thread is running without RT scheduling, which is the one you get pxh shell from. That doesn't matter.

CPU isolating will benefit, but not the critical part. PX4 will fall to meet the scheduling requirement when memory is almost full and page swap happens. I haven' t found a proper way to deal with this case.

https://man7.org/linux/man-pages/man2/mlock.2.html

[dagar]

Which raspberry pi is this? The cpu usage is significantly higher than what I'm seeing.

[blazczak]

I'm staging and optimizing this on one of my RPi 2s for now, I didn't want to risk losing one of my 4s or 3s in a spectacular crash until I'm satisfied that this code labeled as experimental works and is reliable.

[Ncerzzk]

The issue CPU usage is so high may be the same with #19358 because of the ring in schedule link list.

The other hand, I think the .2 and .3 would bring benefit in multi cores platform.