search

LinuxCNC / linuxcnc

LinuxCNC controls CNC machines. It can drive milling machines, lathes, 3d printers, laser cutters, plasma cutters, robot arms, hexapods, and more.
http://linuxcnc.org/
GNU General Public License v2.0
1.77k stars 1.14k forks source link

rtapi_app silently fails, when CONFIG_RT_GROUP_SCHED=y and process is not in root cgroup (just default systemd) #2821

Closed cepelinas9000 closed 8 months ago

cepelinas9000 commented 8 months ago

After digging it looks like is not new problem, but at least some warning should be printed on failure.

Here are the steps I follow to reproduce the issue:

  1. compile realtime kernel with CONFIG_RT_GROUP_SCHED=y
  2. boot that kernel
  3. start latency-test or any realtime program

This is what I expected to happen:

On realtime thread creation failure (not sure, but look like 'rtapi_task_start' function does heavy lifting) error message should be printed. Ideally with pointers to where start digging. (currently it is cgroup problem as RT time need to be given explicitly: https://www.kernel.org/doc/Documentation/scheduler/sched-rt-group.txt).

This is what happened instead:

latency-test starts and print all values "0"

It worked properly before this:

when kernel compiled CONFIG_RT_GROUP_SCHED=n everything works as expected.

Information about my hardware and software:

andypugh commented 8 months ago

Not related to this issue, but please update to the 2.9.2 release of LinuxCNC. There are bugs in the version in Debian and we are not able to update the version on their servers.

Would you expect CONFIG_RT_GROUP_SCHED=y to work? I actually don't know anything about the config switches for preempt-rt, we only test with the released kernel packages.

Do you have any insight into how we might detect the problem inside rtapi? If you are compiling your own kernels (and have a kernel to test with which shows the problem) then you might be in the best position to find the fix.

andypugh commented 8 months ago

A further thought.... If you compiled LinuxCNC from source, did you perform the "sudo make setuid" step? (I am not 100% sure if this is needed for preempt-rt, it might be a carry-over from RTAI)

cepelinas9000 commented 8 months ago

I compiled from source and did "sudo make setuid" to be sure.

I don't expect to work out of the box. It just having log message is enough. As now with CONFIG_RT_GROUP_SCHED=y, you have this: zeros Only clue is when you strace process and see that pthread_set_scheduler FIFO returns "Operation not permitted":

orangepi@orangepi5:~# RTAPI_UID=1000 RTAPI_FIFO_PATH=/home/orangepi/.rtapi_fifo strace  rtapi_app load threads name1=fast period1=25000 name2=slow period2=1000000

...a lot of text...
clone(child_stack=0x7fb7b3ea20, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[3365], tls=0x7fb7b3f8a0, child_tidptr=0x7fb7b3f230) = 3365
sched_setaffinity(3365, 128, [7])       = 0
sched_setscheduler(3365, SCHED_FIFO, [98]) = -1 EPERM (Operation not permitted)
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
futex(0x7fb7b3f578, FUTEX_WAKE_PRIVATE, 1) = 1
setfsuid(0)                             = 1000
setfsuid(1000)                          = 0
sched_get_priority_max(SCHED_FIFO)      = 99
sched_get_priority_min(SCHED_FIFO)      = 1
sched_get_priority_max(SCHED_FIFO)      = 99
sched_get_priority_min(SCHED_FIFO)      = 1
sched_get_priority_min(SCHED_FIFO)      = 1
sched_get_priority_max(SCHED_FIFO)      = 99
sched_get_priority_min(SCHED_FIFO)      = 1
sched_get_priority_max(SCHED_FIFO)      = 99
sched_get_priority_min(SCHED_FIFO)      = 1
openat(AT_FDCWD, "/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 5
read(5, "0-7\n", 1024)                  = 4
close(5)                                = 0
sched_get_priority_min(SCHED_FIFO)      = 1
sched_get_priority_max(SCHED_FIFO)      = 99
rt_sigprocmask(SIG_BLOCK, ~[], [], 8)   = 0
clone(child_stack=0x7fb7b3ea20, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[3366], tls=0x7fb7b3f8a0, child_tidptr=0x7fb7b3f230) = 3366
sched_setaffinity(3366, 128, [7])       = 0
sched_setscheduler(3366, SCHED_FIFO, [97]) = -1 EPERM (Operation not permitted)
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
futex(0x7fb7b3f578, FUTEX_WAKE_PRIVATE, 1) = 1
setfsuid(0)                             = 1000
setfsuid(1000)                          = 0
accept(3,

Ideally i expecting in console log something like this:

orangepi@orangepi5:~$ latency-test 
Note: Using POSIX realtime
Error: pthread_set_scheduler failed, did you set limits.conf or your process cgroup cpu.rt_runtime_us variable (see kernel Documentation/scheduler/sched-rt-group.txt)

After little kernel patching[1], it is possible to manually allow rt tasks:

# some bash commands
root@orangepi5:/sys/fs/cgroup/user.slice# echo "950000" > cpu.rt_runtime_us
root@orangepi5:/sys/fs/cgroup/user.slice# cd user-1000.slice/
root@orangepi5:/sys/fs/cgroup/user.slice/user-1000.slice# echo "950000" > cpu.rt_runtime_us
root@orangepi5:/sys/fs/cgroup/user.slice/user-1000.slice# cd session-5.scope/
root@orangepi5:/sys/fs/cgroup/user.slice/user-1000.slice# echo "950000" > cpu.rt_runtime_us

ctrl-d

orangepi@orangepi5:~$ latency-test

working

[1] patch: When using cgroupv2 with 6.6.0 kernel (6.7-rc7 needs to), you need force rt_runtime_us files appear in hierarchy:

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 011002691..ae97a86c7 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -11559,6 +11559,17 @@ static struct cftype cpu_files[] = {
                .write = cpu_uclamp_max_write,
        },
 #endif
+        {
+                .name = "rt_runtime_us",
+                .read_s64 = cpu_rt_runtime_read,
+                .write_s64 = cpu_rt_runtime_write,
+        },
+        {
+                .name = "rt_period_us",
+                .read_u64 = cpu_rt_period_read_uint,
+                .write_u64 = cpu_rt_period_write_uint,
+        },
+
        { }     /* terminate */
 };

note: this diff is for 6.6.0-rt15+ (6.6.0 kernel with rt15 patch)

cepelinas9000 commented 8 months ago

Taking more time I see there is Merge request https://github.com/LinuxCNC/linuxcnc/pull/2808 which tries handle pthread error, but there is bug in this request (as pthread function do not set errno variable).

As for me, it just enough to have logged error, for example adding extra logging in rtapi_task_start :

diff --git a/src/rtapi/uspace_rtapi_app.cc b/src/rtapi/uspace_rtapi_app.cc
index 440727b17c..ed07367dbb 100644
--- a/src/rtapi/uspace_rtapi_app.cc
+++ b/src/rtapi/uspace_rtapi_app.cc
@@ -1028,16 +1028,18 @@ int Posix::task_start(int task_id, unsigned long int period_nsec)
   int nprocs = sysconf( _SC_NPROCESSORS_ONLN );

   pthread_attr_t attr;
-  if(pthread_attr_init(&attr) < 0)
-      return -errno;
-  if(pthread_attr_setstacksize(&attr, task->stacksize) < 0)
-      return -errno;
-  if(pthread_attr_setschedpolicy(&attr, policy) < 0)
-      return -errno;
-  if(pthread_attr_setschedparam(&attr, &param) < 0)
-      return -errno;
-  if(pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED) < 0)
-      return -errno;
+  int ret;
+
+  if ((ret = pthread_attr_init(&attr)) != 0)
+      return -ret;
+  if ((ret = pthread_attr_setstacksize(&attr, task->stacksize)) != 0)
+      return -ret;
+  if((ret = pthread_attr_setschedpolicy(&attr, policy)) != 0)
+      return -ret;
+  if((ret = pthread_attr_setschedparam(&attr, &param)) != 0)
+      return -ret;
+  if((ret = pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED)) != 0)
+      return -ret;
   if(nprocs > 1) {
       const static int rt_cpu_number = find_rt_cpu_number();
       if(rt_cpu_number != -1) {
@@ -1052,8 +1054,8 @@ int Posix::task_start(int task_id, unsigned long int period_nsec)
                return -errno;
       }
   }
-  if(pthread_create(&task->thr, &attr, &wrapper, reinterpret_cast<void*>(task)) < 0)
-      return -errno;
+  if((ret=pthread_create(&task->thr, &attr, &wrapper, reinterpret_cast<void*>(task))) != 0)
+      return -ret;

   return 0;
 }
@@ -1215,7 +1217,13 @@ int rtapi_task_delete(int id) {

 int rtapi_task_start(int task_id, unsigned long period_nsec)
 {
-    return App().task_start(task_id, period_nsec);
+    int ret = App().task_start(task_id, period_nsec);
+    
+    if (ret != 0){
+   errno = -ret;           // make perror happy
+   perror("FAILED to start rtapi_task, possible causes is missing rt priorities in limits.conf or process cgroup cpu.rt_runtime_us variable not allowing realtime tasks (see kernel Documentation/scheduler/sched-rt-group.txt)");
+    }
+    return ret;
 }

 int rtapi_task_pause(int task_id)

In console looks like this: Untitled3

andypugh commented 8 months ago

If you attach .patch to the link in the referenced patch then you get a file that can be applied on your system to see if it has the correct effect.

I see now that you have already done that. Does this now address your concerns?

cepelinas9000 commented 8 months ago

I see, that the main culprit was wrong error pthread functions error handling. As from my side problem is resolved ( https://github.com/LinuxCNC/linuxcnc/pull/2808/files#diff-68dd48d8bdbf08a8ece3857b0ea4e06c12c57dfed2ad85764cdca5b98bd8d037R1223 ), just there could be some explanation why it can fail for root user.