Tracing tools for ROS 2.
ros2_tracing
provides tracing instrumentation for the core ROS 2 packages.
It also provides tools to configure tracing through a launch action and a ros2
CLI command.
ros2_tracing
currently only supports the LTTng tracer.
Consequently, it currently only supports Linux.
Note: make sure to use the right branch, depending on the ROS 2 distro: use rolling
for Rolling, galactic
for Galactic, etc.
Read the ros2_tracing
paper!
If you use or refer to ros2_tracing
, please cite:
C. Bédard, I. Lütkebohle, and M. Dagenais, "ros2_tracing: Multipurpose Low-Overhead Framework for Real-Time Tracing of ROS 2," IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 6511–6518, 2022.
This other paper leverages ros2_tracing
to analyze and visualize the flow of messages across distributed ROS 2 systems:
C. Bédard, P.-Y. Lajoie, G. Beltrame, and M. Dagenais, "Message Flow Analysis with Complex Causal Links for Distributed ROS 2 Systems," Robotics and Autonomous Systems, vol. 161, p. 104361, 2023.
Finally, check out the following presentations:
Starting from ROS 2 Iron Irwini, the LTTng tracer is a ROS 2 dependency.
Therefore, ROS 2 can be traced out-of-the-box on Linux; this package does not need to be re-built.
The following rmw
implementations are supported:
rmw_connextdds
rmw_cyclonedds_cpp
rmw_fastrtps_cpp
rmw_fastrtps_dynamic_cpp
To make sure that the instrumentation and tracepoints are available:
$ source /opt/ros/rolling/setup.bash # With a binary install
$ source ./install/setup.bash # When building from source
$ ros2 run tracetools status
Tracing enabled
A ROS 2 installation only includes the LTTng userspace tracer (LTTng-UST), which is all that is needed to trace ROS 2. To trace the Linux kernel, the LTTng kernel tracer must be installed separately:
$ sudo apt-get update
$ sudo apt-get install lttng-modules-dkms
For more information about LTTng, refer to its documentation.
To build and remove all instrumentation, use TRACETOOLS_DISABLED
:
$ colcon build --cmake-args -DTRACETOOLS_DISABLED=ON
This will remove all instrumentation from the core ROS 2 packages, and thus they will not depend on or link against the shared library provided by the tracetools
package.
This also means that LTTng is not required at build-time or at runtime.
Alternatively, to only exclude the actual tracepoints, use TRACETOOLS_TRACEPOINTS_EXCLUDED
:
$ colcon build --packages-select tracetools --cmake-clean-cache --cmake-args -DTRACETOOLS_TRACEPOINTS_EXCLUDED=ON
This will keep the instrumentation but remove all tracepoints.
This also means that LTTng is not required at build-time or at runtime.
This option can be useful, since tracepoints can be added back in or removed by simply replacing/re-building the shared library provided by the tracetools
package.
By default, trace data will not be generated, and thus these packages will have virtually no impact on execution. LTTng has to be configured for tracing. The packages in this repo provide two options: a command and a launch file action.
Note: tracing must be started before the application is launched. Metadata is recorded during the initialization phase of the application. This metadata is needed to understand the rest of the trace data, so if tracing is started after the application started executing, then the trace data might be unusable. For more information, refer to the design document. The launch file action is designed to automatically start tracing before the application launches.
The tracing directory can be configured using command/launch action parameters, or through environment variables with the following logic:
$ROS_TRACE_DIR
if ROS_TRACE_DIR
is set and not empty.$ROS_HOME/tracing
, using ~/.ros
for ROS_HOME
if not set or if empty.Additionally, if you're using kernel tracing with a non-root user, make sure that the tracing
group exists and that your user is added to it.
# Create group if it doesn't exist
$ sudo groupadd -r tracing
# Add user to the group
$ sudo usermod -aG tracing $USER
The first option is to use the ros2 trace
command.
$ ros2 trace
By default, it will enable all ROS 2 tracepoints.
The trace will be written to ~/.ros/tracing/session-YYYYMMDDHHMMSS
.
Run the command with -h
for more information.
The ros2 trace
command requires user interaction to start and then stop tracing.
To trace without user interaction (e.g., in scripts), or for finer-grained tracing control, the following sub-commands can be used:
$ ros2 trace start session_name # Configure tracing session and start tracing
$ ros2 trace pause session_name # Pause tracing after starting
$ ros2 trace resume session_name # Resume tracing after pausing
$ ros2 trace stop session_name # Stop tracing after starting or resuming
Run each command with -h
for more information.
You must install the kernel tracer if you want to enable kernel events (using the -k
/--kernel-events
option) or syscalls (using the --syscalls
option).
If you have installed the kernel tracer, use kernel tracing, and still encounter an error here, make sure to add your user to the tracing
group.
Another option is to use the Trace
action in a Python, XML, or YAML launch file along with your Node
action(s).
This way, tracing automatically starts when launching the launch file and ends when it exits or when terminated.
$ ros2 launch tracetools_launch example.launch.py
The Trace
action will also set the LD_PRELOAD
environment to preload LTTng's userspace tracing helper(s) if the corresponding event(s) are enabled.
For more information, see this example launch file and the Trace
action.
You must install the kernel tracer if you want to enable kernel events (events_kernel
in Python, events-kernel
in XML or YAML) or syscalls (syscalls
in Python, XML, or YAML).
If you have installed the kernel tracer, use kernel tracing, and still encounter an error here, make sure to add your user to the tracing
group.
See the design document.
LTTng-UST, the current default userspace tracer used for tracing ROS 2, was designed for real-time production applications. It is a low-overhead tracer with many important real-time compatible features:
However, some settings need to be tuned for it to be fully real-time safe and for performance to be optimal for your use-case:
[^rt-1]: this setting cannot currently be set through the Trace
launch file action or the ros2 trace
command, see #20
For further reading:
The LTTng kernel tracer has a similar implementation, but is separate from the userspace tracer.
Package containing liblttng-ctl
Python bindings.
Package containing a ros2cli
extension to enable tracing.
Library to support instrumenting ROS packages, including core packages.
This package claims to be in the Quality Level 1 category, see the Quality Declaration for more details.
See the API documentation.
Package containing tools to enable tracing through launch files.
Package containing tools to read traces.
Package containing tools for tracing-related tests.
Package containing tools to enable tracing.
Package containing system tests for ros2trace
.
Package containing unit and system tests for tracetools
.
Package containing system tests for tracetools_launch
.
See tracetools_analysis
.