unable to install lttng on rt linux kernel

[flynneva]

so I recently ran into this problem while trying to setup ros2_tracing to track system performance. essentially I was unable to sudo apt install lttng-modules-dkms as it said the RT kernel was not officially supported.

I did some digging online and came across this thread that looks to be exactly the same issue.

It goes on to suggest a solution of compiling lttng from source....which ideally i would like to avoid.

As lttng is a pretty universal tracer tool I was wondering if an alternative proposal could happen where we add it to the Dockerfile here so that we can compile it right into the kernel for use later on.

I found this README for lttng-modules that looks like it has instructions to add it directly to a kernel build.

lttng-modules-README.md

### Kernel built-in support

It is also possible to build these modules as part of a kernel image. Simply
run the [`scripts/built-in.sh`](scripts/built-in.sh) script with the path to
your kernel source directory as an argument.  It will symlink the
lttng-modules directory in the kernel sources and add an include in the kernel
Makefile.

Then configure your kernel as usual and enable the `CONFIG_LTTNG` option.

### Required kernel config options

Make sure your target kernel has the following config options enabled:

  - `CONFIG_MODULES`: loadable module support (not strictly required
     when built into the kernel),
  - `CONFIG_KALLSYMS`: see files in [`wrapper`](wrapper); this is
     necessary until the few required missing symbols are exported to GPL
     modules from mainline,
  - `CONFIG_HIGH_RES_TIMERS`: needed for LTTng 2.x clock source,
  - `CONFIG_TRACEPOINTS`: kernel tracepoint instrumentation
     (enabled as a side-effect of any of the perf/ftrace/blktrace
     instrumentation features).
  - `CONFIG_KPROBES` (5.7+): use kallsyms for kernel 5.7 and newer.

[flynneva]

would adding lttng to the kernel build process be an acceptable enhancement here? or is it not acceptable to add it to the default kernel build?

[flynneva]

also see this issue over at the raspberrypi/linux repo

[razr]

@flynneva yes, we should do it. It was not clear to me whether RT and lttng work together, but apparently they do. Do you want to give it a try and provide a PR?

[christophebedard]

I was planning on trying do this soon, but if you can figure out how to do it it'd be great!

[flynneva]

so I think I got this working but I wont know for sure until the build fully completes.

you can see what I did in #18 in the Dockerfile to get it added to the kernel. the trickiest bit was figuring out that we also needed the scripts/rt-patch-version.sh which I only guessed at since it wasnt anywhere in the docs that I could find.

what makes me think I have it right is during the build, right at the beginning it prompted me a few questions specifically about LTTng and its configuration...so fingers crossed.

[flynneva]

would love to have someone more familiar with lttng and ros2_tracing check my work over on #18, however I think I've got it working following the normal kernel build instructions from the README.

the only change to the process that i did was i skipped sudo apt install lttng-dkms-modules since we built it into the kernel. everything seems to be working OK...however I dont fully understand if what im seeing in the tracing metadata is "right" or not.

maybe @christophebedard @razr or @LanderU knows more and can understand if things look OK?

[christophebedard]

I think it looks good! The metadata file always looks a bit weird (it's not 100% text I think?). You can check the CTF spec if you want (https://diamon.org/ctf/), but it's probably not worth it :stuck_out_tongue_closed_eyes:

You could simply take a look at the userspace and kernel traces separately to make sure stuff was recorded:

$ babeltrace /home/ubuntu/.ros/tracing/my-tracing-session/kernel
$ babeltrace /home/ubuntu/.ros/tracing/my-tracing-session/ust

With the default setup, the kernel trace will probably have a lot more events, which you can count by piping the babeltrace output to wc -l.

[flynneva]

awesome to hear @christophebedard, thanks! yes I removed a bunch of gibberish at the beginning (non-text stuff) when copying to github so thats probably what you were talking about.

[flynneva]

looking good:

babeltrace /home/ubuntu/.ros/tracing/my-tracing-session/kernel | wc -l
63546

babeltrace /home/ubuntu/.ros/tracing/my-tracing-session/ust | wc -l
130

thanks for the tips!