search

thin-edge / thin-edge.io

The open edge framework for lightweight IoT devices
https://thin-edge.io
Apache License 2.0
223 stars 55 forks source link

avoid using colons in tedge-agent command log file names #3165

Open reubenmiller opened 1 month ago

reubenmiller commented 1 month ago

Is your feature improvement request related to a problem? Please describe.

The current tedge-agent log file names under /var/log/tedge/agent/*.log use the colon (:) character which results in slight annoyances when selecting files names on the console, or transferring the log files to NTFS storage mediums.

This mostly affects the log files created when the command id is based on a timestamp (e.g. RFC3339).

Below shows an example of some of the files which make use of colons (:):

$ ls -ltr /var/log/tedge/agent

total 80
-rw-r--r-- 1 tedge tedge 14652 Oct  7 06:47 workflow-software_list-c8y-mapper-2024-10-07T06:47:36.491494387Z.log
-rw-r--r-- 1 tedge tedge 10185 Oct  7 06:47 workflow-software_list-c8y-mapper-2024-10-07T06:47:40.881845183Z.log
-rw-r--r-- 1 tedge tedge 10185 Oct  7 06:47 workflow-software_list-c8y-mapper-2024-10-07T06:47:41.047587459Z.log
-rw-r--r-- 1 tedge tedge   519 Oct  7 06:49 c8y_RemoteAccessConnect-2024-10-07T06:49:52.706238934Z.log
-rw-r--r-- 1 tedge tedge 10170 Oct  7 06:53 workflow-software_list-c8y-mapper-2024-10-07T065303.502675425Z.log
-rw-r--r-- 1 tedge tedge 10173 Oct  7 06:53 workflow-software_list-c8y-mapper-2024-10-07T065303.578293335Z.log
-rw-r--r-- 1 tedge tedge   519 Oct  7 06:53 c8y_RemoteAccessConnect-2024-10-07T06:53:08.020012572Z.log
-rw-r--r-- 1 tedge tedge  2487 Oct  7 07:43 workflow-log_upload-c8y-mapper-40290121.log
-rw-r--r-- 1 tedge tedge  2433 Oct  7 07:43 workflow-log_upload-c8y-mapper-40290123.log

See the Additional Context section for more examples where this incompatibility was discovered.

Describe the solution you'd like

Avoid using colons (:) in the log file names to improve portability and usability of the log files.

It should be noted that it might be also worthwhile changing the tedge MQTT command id (which is where the filename is being derived from) to also avoid using colons (:) so that the command id and filename remain in sync.

There are a few options to consider how to format the log file names:

Describe alternatives you've considered

Additional context

Using Github Action: upload artifact action

The following error was printed whilst trying to upload the log files collected from a thin-edge.io system test run, and upload them to the Github Runner.

The path for one of the files in artifact is not valid: /logs/child01/agent/workflow-software_list-c8y-mapper-2024-10-03T10:27:52.189032064Z.log. Contains the following character:  Colon :

Invalid characters include:  Double quote ", Colon :, Less than <, Greater than >, Vertical bar |, Asterisk *, Question mark ?, Carriage return \r, Line feed \n

The following characters are not allowed in files that are uploaded due to limitations with certain file systems such as NTFS. To maintain file system agnostic behavior, these characters are intentionally not allowed to prevent potential problems with downloads on different file systems.

Below also shows the aforementioned forbidden filepath characters as listed on the Microsoft documentation:

Using iTerm - MacOS Terminal

Selecting the file on the command line is annoying because the default setting in iTerm is to treat the : as a word separator, so double clicking on the file name will only highly before or after the : character. Below shows a screenshot when trying to double click the filename:

image
didier-wenzek commented 1 month ago
reubenmiller commented 1 month ago
  • I'm against the idea to use url encoding. We just need to use another separator instead of columns :.

Though this raises interesting questions about a similar scenario where the external identity is used to store the child device supported operations (under /etc/tedge/operations/c8y/<external_identity>), where thin-edge.io also uses colons (:).