search

COVESA / dlt-viewer

Diagnostic Log and Trace viewing program
Other
425 stars 240 forks source link

CSV command line export encoding -u flag no effect? #384

Closed suiluj closed 6 months ago

suiluj commented 11 months ago

Hello I am trying to convert dlt files to csv via the command line interface.

Should the flag -u have an effect on the encoding of the csv file?

At first I tried this conversion command and it works perfectly for some dlt files:

dlt-viewer.exe -s -csv -c <input.dlt> <output.csv>

Most of the time it works and I get a csv file that I can work with.

But for some csv files I get encoding errors while reading file. My guess is that there is some strange logging output in some payload messages.

To fix the encoding problems i tried with the -u flag:

dlt-viewer.exe -s -csv -c <input.dlt> <output.csv>

But the problem stayed the same.

For the moment my solution is to replace encoding errors while reading the file but of course it would be better to have a "perfect" csv file in the first place. :)

Perhaps someone had a similar problem and found a solution. Thanks in advance.

alexmucde commented 11 months ago

Hi @suiluj. Could it be that you have corrupted messages in your DLT file or some Linefeeds/Cariage Returns in your AppIds/CtxIds. There exists a fix already, that fixed the issue: Remove Linefeeds and Cariage Returns in CSV Export of ECUId, AppId and CtxId. Please check if this helps.

suiluj commented 11 months ago

Hi @alexmucde yes I noticed that there was a pull request for a fix.

But in my case it seems that it is part of the payload column

When I tried to load the exported csv with python pandas or duckdb I got an error that said there is an error in the payload column.

Ich checked the row but did not notice any strange characters in dlt viewers.

alexmucde commented 6 months ago

@suiluj So could be another issue in your payload. Please check the source code. Perhaps some characters must be removed or replaced.

https://github.com/COVESA/dlt-viewer/blob/5ddd64202f2fb26b19ff4537f210de080cec4755/src/dltexporter.cpp#L53

Please provide more details which characters could cause the issues. Best would be to provide a patch file.

suiluj commented 6 months ago

hi @alexmucde

I just kept my solution with the file read and "replace" on error.

(I use python for processing the generated csv file) https://docs.python.org/3/library/functions.html#open

Since then I never noticed any problems in my workflow so we can close this issue.