Better log path configuration

[janhumble]

Some devices clear the /dev/log path upon reboot. All thin-edge services fail to start because of this.

1. the initial installation script and services log handling should be more resilient to missing log path structures. If an operator or system function deletes the log folder, there's no longer a way to communicate with the device from Cumulocity as services fail to start. This is a major issue on the field.
2. Documentation should better present how to configure log paths and troubleshoot current limitations.
3. Log path configurations should be easy to apply and work with child device representations.

[reubenmiller]

Some devices clear the /dev/log path upon reboot. All thin-edge services fail to start because of this.

1. the initial installation script and services log handling should be more resilient to missing log path structures. If an operator or system function deletes the log folder, there's no longer a way to communicate with the device from Cumulocity as services fail to start. This is a major issue on the field.
2. Documentation should better present how to configure log paths and troubleshoot current limitations.
3. Log path configurations should be easy to apply and work with child device representations.

@janhumble Thanks for the ticket. In the future can you please use a ticket template. It just helps organize the information a bit better.

[reubenmiller]

Though I have a few questions regarding the description:

Some devices clear the /dev/log path upon reboot. All thin-edge services fail to start because of this.

What do you mean by /dev/log path. Do you just mean all log files and folder required by thin-edge.io? Do you want to move the log path to a location which is persisted across reboots, or are you ok with the data under there not being persisted?

Log path configurations should be easy to apply and work with child device representations.

Are you just asking that the location where the configuration files for child devices are stored be configurable?

[jmoo900]

Hi Reuben. Just following up on this ticket to keep it active.

I believe the path is actually the /var/log path and is designated as volatile in latest version of centos. My understanding of the issue is that because of this classification, the path gets removed with each reboot. This is resulting in the agent failing to start because it can't find the log path. I recommended the customer use the "tedge config set logs.path" command to configure a different path that isn't volatile. Will this resolve the recurring issue?

[reubenmiller]

tedge config set logs.path

Yes changing that to a non-volatile path will help, though if /var/tedge is also not persistent then there will be some other problems as this folder is used for caching purposes, so tedge-agent will fail if it does not exist. This change would be covered by this ticket.

Though in the meantime there are other ways around solving this problem, but it depends on how quickly you need the solution. If you can't wait, then you could create a new systemd service file which runs a command to recreate all of the required folders, and you can configure the service to run before thin-edge's components.

Using a basic systemd service, we can make sure custom logic runs before the thin-edge components start.

Instructions

Create a file with the following contents:

File: /etc/systemd/system/tedge-init-workaround.service

[Unit]
Description=tedge init workaround
Before=tedge-agent.service
Before=tedge-mapper-c8y.service
Before=c8y-configuration-plugin.service
Before=c8y-log-plugin.service

[Service]
Type=simple
ExecStart=/bin/bash -c "tedge --init; tedge-agent --init; tedge-mapper --init c8y; c8y-configuration-plugin --init; c8y-log-plugin --init"

[Install]
WantedBy=multi-user.target

Then enable the service to start on bootup.

sudo systemctl daemon-reload
sudo systemctl enable tedge-init-workaround.service

Then reboot your device to check if everything works. After it boots up you can check if the tedge-init-workaround service ran successfully (ignore the Connection Error I/O: Connection refused (os error 111) messages)

systemctl status tedge-init-workaround.service

Example output of the init service, the service should not be running anymore but it should have been successful.

systemctl status tedge-init-workaround.service
● tedge-init-workaround.service - tedge init workaround
     Loaded: loaded (/etc/systemd/system/tedge-init-workaround.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Thu 2023-03-09 20:49:06 CET; 2min 34s ago
    Process: 399 ExecStart=/bin/bash -c tedge --init; tedge-agent --init; tedge-mapper --init c8y; c8y-configuration-plugin --init; c8y-log-plugin --init (code=exited, status=0/SUCCESS)
   Main PID: 399 (code=exited, status=0/SUCCESS)
        CPU: 131ms

References

You can read more about systemd and all of its possibilities from their documentation

[reubenmiller]

I've created another ticket to cover the configurable http server root folder (/var/tedge/) setting. https://github.com/thin-edge/thin-edge.io/issues/1804

[reubenmiller]

The paths are now configurable in 0.10.0. Checkout the documentation for details on which paths can be configured and how.