Explore Linux system logging and how we will store and handle Linux system logs on the OBC, as well as how we plan on accessing them from the GS.

[dheadrick1618]

Description

Task is to get logging working on the FSW. Need to determine how the system should log, and how this could be accessed by the ground station.

Deliverables

• [x] Determine what the industry convention is for Linux logging (Just moving stderr to a file?)
• [ ] Find out if we can use the tools that was found in research in ex-alta.
• [ ] Make the architecture plan for logging
• [ ] Offload the research in this card in to a markdown document.
• [ ] #12

Research

Typical Linux use case

• Kernel logging vs User logging
  • User space vs kernel space.
  • Kernel has a ring buffer that holds log messages. It's a circular buffer. Starts before user logging. Accessible via dmesg.
  • The terminal, and therefore the user space is able to access the circle buffer by reading a file called /dev/kmsg. This file has access to the ring buffer structure that holds the log.
  • The ring buffer will then dump the content into /proc/kmsg.
  • /dev/kmsg acts like an entry point to the ring buffer, since you are able to both read and write to it.
  • /proc/kmsg is only for reading since it is not an entry point to the ring buffer.
  • User space logging is based on the syslog protocol.
  • This protocol defines how messages are formatted for an application to send logs.
  • Two new terms: facilities and priorities.
    • Facility levels - Used to determine which part of the system sent the log. There are at least 23 facilities, and they include things from kernel and mail to clock and scheduling. A few of the levels are preserved for custom programs.
    • Priorities - Shows the severity of the log message. There are 8 levels, with the lowest value being the biggest emergency.
  • Architecture:
    • Originator - The syslog client. Sent the syslog formatted message over the network, or to the correct application.
    • Relay - used to forward messages over a network. Could add more to the message.
    • Collector - The syslog server. Collects and stores (and maybe view) the log.
  • Syslog is a client server architecture.
  • Linux local logging
  • You can log using the same architecture as syslog.
  • client applications are originators
  • no need for a relay, (unless we are transmitting back to ground station?)
  • Collectors are daemons that are listening on a certain socket.
  • Supposedly, in the Linux system, logs are stored in the /var/log directory.
  • The are two systems in the Linux operating system that collects logs. rsyslog and systemd-journal.
    • rsyslog makes permanent log in the /var/log directory
    • Listens to the syslog socket for new logs.
    • There are rules that controls which logs should be stored.
    • systemd-journal keeps a temp log unless configured otherwise.
    • A little weirder since the files are stored as binaries instead of a usually text file. Need to use journalctl to access the content.
    • Stored files in the /run/log/journal directory.
    • There is a way to store journal logs into rsyslog by changing the configuration of the rsyslog to take from journal.
    • logger is a built in utility that will send strings to a syslog server.

Embedded readings

printf is expensive, so logging should maybe be deferred.

• An operating system called Zephyr uses deferred logging.
• Move the logging operation elsewhere. The logging client does not need to concern itself with string manipulation since it might be responsible for time critical operations.

The thing is, we are working with Linux, so we should be able to leverage the socket design paradigm to let the operating system do the scheduling work for us.

Figures

Value	Severity	Keyword
0	Emergency	emerg
1	Alert	alert
2	Critical	crit
3	Error	err
4	Warning	warning
5	Notice	notice
6	Informational	info
7	Debug	debug

The table above is the list of priorities in the syslog standard.

Resources

• Top result on how logging works on Linux
• Embedded ninja post on embedded logging
• 2008 Stack Overflow answer that says that syslog was good for embedded linux

[dheadrick1618]

Great work! I think we should include the definition of each log severity for clarification, such as:

I think the next step is to make a small working example / demonstration program that scrapes various logs and consolidates them, so a simple GS command can request all this log info in a simple manner. Thoughts?

[waridh]

Yeah, I agree. But I have one question. Should we try to reuse syslog before moving to making a custom implementation of the log server process?