[Draft] Added syslog handler for launch log

[firesurfer]

This PR adds initial support for writing to the syslog. It goes hand in hand with: https://github.com/ros2/rcl_logging/pull/105 In combination it allows to write the whole log a ros2 application creates to the syslog. Both PRs use 'local1' as facility which allows to redirect the whole log into a seperate log file or even a remote logging server without the need for additional configuration inside the ros2 logging system. (Everything can be configured in rsyslog.conf)

And also answers my question from #736

Why do we need it

In enterprise environments it can be desirable to store the whole application log on a remote logging server. Utilizing rsyslog for providing this functionality is an easy and established way.

Just grabbing /rosout won't suffice as we cannot grab all parts of the log.

What is missing so far:

1. Allow configuring the SysLogHandler (address, facility)
2. Allow to disable/enable the SysLogHandler
3. Implement custom formatter / do not reuse the file_formatter

How to test it:

I redirected the 'local1' facility to /var/log/local1.log. This allows testing without spamming into your normal syslog.

sudo touch /var/log/local1.log
sudo chown syslog /var/log/local1.log

Edit /etc/rsyslog.d/50-default.conf And add the line:

local1.*                        -/var/log/local1.log

Restart rsyslog

sudo systemctl restart rsyslog.service 

Afterwards you should see the most basic output written into the file. Eg:

Oct 11 08:33:01 my_pc 1697005981.4697814 [INFO] [launch]: All log files can be found below /home/my_user/.ros/log/2023-10-11-08-33-01-469156-my_pc-2027270

Some additional notes

The current implementation grabs on purpose only the parts of the log which can't be grabbed via rcl_logging (https://github.com/ros2/rcl_logging/pull/105). Otherwise some log messages would occur twice in the syslog.

[firesurfer]

Just a quick push, as I would really like to progress on these patches :)

[sloretz]

@clalancette assigned you to provide thoughts on Windows 🧇

[firesurfer]

@clalancette As I already wrote in https://github.com/ros2/rcl_logging/pull/105 we could use the Event Log on Windows instead: https://docs.python.org/3/library/logging.handlers.html#nteventloghandler

Additionally if desired by the user we could support writing to a remote Syslog server (The python logging handler implementation supports this in contrast to spdlog)

logging.handlers.SysLogHandler(address=('localhost', SYSLOG_UDP_PORT), facility=LOG_USER, socktype=socket.SOCK_DGRAM)

[130s]

without the need for additional configuration inside the ros2 logging system. (Everything can be configured in rsyslog.conf)

I found this interesting. Would you elaborate the decision process?

Personally I prefer configuration of ROS itself (i.e. the destination of log that ROS generates, in this context) to be done in ROS level. But I want to hear other thoughts.

[firesurfer]

@130s We use ROS for multiple industrial machines. At some point a central log server will be necessary and is also common industry practice.

See the "Why do we need it section"

Personally I prefer configuration of ROS itself (i.e. the destination of log that ROS generates, in this context) to be done in ROS level. But I want to hear other thoughts.

This is still the case. It is just as ROS level syslog is selected as sink. The way it is then further processed is up the setup of the syslog deamon.

[130s]

I guess I couldn't make the intent of my question clear enough. Let me retry. I did NOT ask why you want ROS to send logs to syslog (I do like aggregating all logs, not just the ones ROS generates, at one location e.g. syslog or any central place).

Instead, what I asked is why you chose rsyslog.conf to be the place to configure ROS, so that rcl_logger sends log to syslog.

Everything can be configured in rsyslog.conf

AFAIK rsyslog.conf is outside of ROS. Because I prefer configuring ROS to be done in ROS level, I want a way in ROS (e.g. launch, node) to tell ROS "hey, send logs to syslog".

I'm asking this as a non-blocking question for now.

[firesurfer]

Sorry that I understood your question wrong.

1. It is the default common logging facility of most linux systems
2. It is way more flexible than what the ROS logging can do
3. Separate application and logging configuration on the system
4. Additionally it is a way to obtain all logs from multiple systems running a ROS application. At the moment just capturing /ros_out is not enough.

AFAIK rsyslog.conf is outside of ROS. Because I prefer configuring ROS to be done in ROS level, I want a way in ROS (e.g. launch, node) to tell ROS "hey, send logs to syslog".

Well thats what this patch is doing. You tell ROS send to syslog. What is syslog: It is in most cases rsyslogd which is configured via rsyslog.conf.

[130s]

What is syslog: It is in most cases rsyslogd which is configured via rsyslog.conf.

On OS-level, sure. What I was saying is that I'd add an option somewhere in ROS, e.g. launch or else, so that I can configure ROS to send log to rsyslogd, without directly touching rsyslog.conf. Intent of that is to avoid customization on "sudo-protected" area as much as possible for the ease of configuration management.

But, that's just my preference, which should be opened separately as a feature enhancement discussion. I saw your https://github.com/ros2/rcl_logging/pull/105#issuecomment-2345337939 and now I understand all these syslog improvement is on hold anyways.