Logging binary UART stream to file not working, only with --hex-output option

[donok1]

Hi,

I'm trying to log raw data from my ublox GPS receiver on my raspberry pi using the UART serial port. Baud rate of 115200 and default 8N1 encoding.

The messages I expect to read should start with B5 62 02 15 in HEX.

I'm using the following command:

grabserial -S -d "/dev/ttyS0" -B 115200 -e 10 -o 'test_%Y-%m-%d_%H:%M:%S.ubx' -Q -v -x

The output, once converter in HEX looks like this:

0000: 62 15 01 49 01 41 66 08 12 0C 01 01 19 0F 76 6A   b¿¿I¿Af¿¿¶¿¿¿¿vj
0010: 77 41 78 00 41 02 68 12 45 00 14 00 00 26 04 03   wAx¿A¿h¿E¿¿¿¿&¿¿
0020: 07 07 00 62 06 76 41 4C 7A 34 07 0A 41 55 0A 44   ¿¿¿b¿vALz4¿¬AU¬D
0030: 00 10 00 00 1D 06 05 08 07 00 5E 26 71 75 41 69   ¿¿¿¿¿¿¿¿¿¿^&quAi
0040: 7D 43 41 42 00 1D 00 00 25 04 03 07 0F 00 6C 77   }CAB¿¿¿¿%¿¿¿¿¿lw
0050: 41 02 01 36 41 50 73 00 05 00 00 20 06 05 08 07   A¿¿6APs¿¿¿¿ ¿¿¿¿
0060: 00 03 2A 2F 40 74 78 41 2D 46 3A 10 41 51 28 45   ¿¿*/@txA-F:¿AQ(E
0070: 00 17 00 00 00 00 19 09 0F 0A 01 00 75 75 6C 77   ¿¿¿¿¿¿¿Δ¿¬¿¿uulw
0080: 41 21 07 64 41 1B 37 02 0C 00 00 23 06 04 08 07   A!¿dA¿7¿¶¿¿#¿¿¿¿
0090: 00 75 79 77 41 32 41 3C 7E 34 02 19 00 00 00 00   ¿uywA2A<~4¿¿¿¿¿¿
00A0: 15 0A 0F 0C 01 00 33 7E 12 29 79 41 3D 42 4B 0C   ¿¬¿¶¿¿3~¿)yA=BK¶
00B0: 1E 41 54 16 44 02 02 00 00 20 06 05 08 07 00 3C   ¿AT¿D¿¿¿¿ ¿¿¿¿¿<
00C0: 4C 12 03 7F 79 41 42 70 62 75 41 02 4C 4D 02 18   L¿¿yABpbuA¿LM¿¿
00D0: 00 00 1C 07 09 08 07 00 0E 7D 45 72 41 67 36 5C   ¿¿¿¿Δ¿¿¿¿}ErAg6\
00E0: 78 41 5F 02 12 00 00 26 05 03 07 07 00 47 32 77   xA_¿¿¿¿&¿¿¿¿¿G2w
00F0: 41 61 41 2C 51 0C 02 0C 06 00 24 1D 06 06 08 07   AaA,Q¶¿¶¿¿$¿¿¿¿¿
0100: 00 6F 75 41 75 41 0C 17 00 1D 03 00 1C 08 07 08   ¿ouAuA¶¿¿¿¿¿¿¿¿¿
0110: 07 00 54 62 01 07 5C 00 08 07 03 16 0F 33 37 05   ¿¿Tb¿¿\¿¿¿¿¿¿37¿
0120: 00 00 00 0B 03 01 0C 66 1F 05 16 47 1C 54 08 00   ¿¿¿¿¿¿¶f¿¿¿G¿T¿¿
0130: 07 00 5F 05 00 00 00 00 10 00 00 00 15 00 00 00   ¿¿_¿¿¿¿¿¿¿¿¿¿¿¿¿
0140: 39 5A 00 00 00 00 0F 6E 00 00 00 00 49 32 00 00   9Z¿¿¿¿¿n¿¿¿¿I2¿¿
0150: 00 00 00 00 00 00 0B 62 01 22 14 00 08 18 0C 00   ¿¿¿¿¿¿¿b¿"¿¿¿¿¶¿
0160: 01 00 00 05 00 00 00 44 02 00 00 3C ->62 02 15<- 01   ¿¿¿¿¿¿¿D¿¿¿<b¿¿¿
0170: 1D 5A 64 3B 4B 01 41 66 08 12 0C 01 01 4E 1A 05   ¿Zd;K¿Af¿¿¶¿¿N¿¿
0180: 16 6A 77 41 4C 70 41 06 6A 12 45 00 14 00 00 26   ¿jwALpA¿j¿E¿¿¿¿&
0190: 04 02 07 07 00 7C 1D 08 06 76 41 71 04 41 2D 00   ¿¿¿¿¿|¿¿¿vAq¿A-¿
01A0: 44 00 10 00 00 1D 06 06 08 07 00 33 31 79 75 41   D¿¿¿¿¿¿¿¿¿¿31yuA
01B0: 41 41 61 42 00 1D 00 00 25 04 03 07 0F 00 2E 71   AAaB¿¿¿¿%¿¿¿¿¿.q
01C0: 77 41 20 6F 36 41 3A 66 00 05 00 00 20 06 06 09   wA o6A:f¿¿¿¿ ¿¿Δ
01D0: 07 00 53 6F 39 74 78 41 64 36 10 41 6D 28 45 00   ¿¿So9txAd6¿Am(E¿
01E0: 17 00 00 00 00 19 09 0F 0A 01 00 49 77 41 3B 4E   ¿¿¿¿¿¿Δ¿¬¿¿IwA;N
01F0: 41 15 37 02 0C 00 00 23 06 04 08 07 00 5C 77 41   A¿7¿¶¿¿#¿¿¿¿¿\wA
0200: 7E 1C 41 18 4E 33 02 19 00 00 00 00 15 0A 0F 0C   ~¿A¿N3¿¿¿¿¿¿¿¬¿¶
0210: 01 00 32 12 21 25 79 41 44 6A 75 1B 41 42 1C 44   ¿¿2¿!%yADju¿AB¿D
0220: 02 02 00 00 20 06 05 08 07 00 62 0A 7F 79 41 51   ¿¿¿¿ ¿¿¿¿¿b¬yAQ
0230: 7A 41 07 5E 4D 02 18 00 00 1C 07 09 08 07 00 73   zA¿^M¿¿¿¿¿¿Δ¿¿¿s
0240: 02 4E 72 41 66 67 78 41 5F 02 12 00 00 26 05 02   ¿NrAfgxA_¿¿¿¿&¿¿
0250: 07 07 00 40 77 41 41 19 48 0C 02 0C 06 00 25 1C   ¿¿¿@wAA¿H¶¿¶¿¿%¿
0260: 07 07 08 07 00 5A 6C 39 77 75 41 08 64 75 41 7C   ¿¿¿¿¿Zl9wuA¿duA|
0270: 17 00 1D 03 00 1C 08 07 08 07 00 62 01 07 5C 00   ¿¿¿¿¿¿¿¿¿¿¿b¿¿\¿
0280: 08 07 03 16 0F 33 37 05 00 00 00 47 17 03 01 0C   ¿¿¿¿¿37¿¿¿¿G¿¿¿¶
0290: 66 1F 05 16 47 1C 54 08 00 07 00 5F 05 00 00 00   f¿¿¿G¿T¿¿¿¿_¿¿¿¿
02A0: 00 09 00 00 00 05 00 00 00 0B 00 00 00 39 5A 00   ¿Δ¿¿¿¿¿¿¿¿¿¿¿9Z¿
02B0: 00 00 00 2C 6E 00 00 00 00 49 32 00 00 00 00 00   ¿¿¿,n¿¿¿¿I2¿¿¿¿¿
02C0: 00 00 00 58 75 62 01 22 14 00 08 18 0C 00 01 00   ¿¿¿Xub¿"¿¿¿¿¶¿¿¿
02D0: 00 05 00 00 00 45 02 00 00 65 59 ->62 02 15<- 01 4C   ¿¿¿¿¿E¿¿¿eYb¿¿¿L
02E0: 01 41 66 08 12 0C 01 01 16 1B 42 6A 77 41 65 5C   ¿Af¿¿¶¿¿¿¿BjwAe\
02F0: 41 18 65 12 45 00 14 00 00 26 04 02 07 07 00 72   A¿e¿E¿¿¿¿&¿¿¿¿¿r
...

The message is not coming up, the B5 character µ (mu) never comes up.

But when I add the --hex-output option, the hex file content seems to show the correct messages:

->B5 62 02 15<- 90 01 1D 5A 64 3B 43 84 01 41 66 08 
12 0C 01 01 66 16 D2 89 7A F4 EC 6A 77 41 6C 45 
7B 49 E3 C3 9E 41 0F 78 12 45 00 14 00 00 F4 FB 
26 05 03 07 07 00 89 0A 4F 46 1F 06 76 41 E4 33 
20 EF 21 EF 9C 41 5F 15 EF 44 00 10 00 00 F4 FB 
1D 06 05 07 07 00 85 04 38 2C 54 F0 75 41 43 26 
6B 59 80 D2 9C 41 50 8A 42 C5 00 1D 00 00 F4 FB 
25 05 03 07 0F 00 2D 29 4C 3F 5B C2 77 41 FA 3A 
E1 06 BF 36 9F 41 7F 52 E6 C4 00 05 00 00 A8 98 
20 06 05 08 07 00 D6 CC 93 CC 59 74 78 41 91 C7 
F3 B0 4B 10 A0 41 3A 79 28 45 00 17 00 00 00 00 
19 09 0F 0A 01 00 F8 16 26 B1 CE 8A 77 41 D4 E8 
B9 C6 C5 ED 9E 41 E1 16 37 C5 02 0C 00 00 F4 FB 
23 06 03 07 07 00 19 5D 6B CE EC 80 77 41 E1 B4 
4C E0 C9 E0 9E 41 10 35 C4 02 19 00 00 00 00 15 
0A 0F 0C 01 00 6E CE 1D D7 38 AC 79 41 B9 93 47 
67 28 DD A0 41 C8 0E CF 44 02 02 00 00 F4 FB 20 
06 05 08 07 00 A8 2F 02 D3 E3 7E 79 41 F2 C5 9E 
DA 60 BF A0 41 7E 4F 4D C5 02 18 00 00 F4 FB 1C 
07 08 08 07 00 C2 7D C5 72 23 A0 72 41 49 0C 6E 
7A 2F 78 98 41 C5 A0 5F C5 02 12 00 00 F4 FB 26 
05 03 07 07 00 5D 63 AC 19 CD 8A 77 41 AC 96 0B 
8A DB B2 97 41 12 48 0C C5 02 0C 06 00 98 21 1C 
07 05 07 07 00 80 DB 81 26 52 F0 75 41 E3 20 61 
7C 79 75 96 41 B9 A0 17 C5 00 1D 03 00 F4 FB 1C 
08 08 08 07 00 D8 CF B5 62 01 07 5C 00 D0 94 8D 
08 E5 07 03 16 0F 33 12 37 05 00 00 00 27 AF DA 
17 03 01 EA 0C CD 66 1F 05 19 A2 47 1C 63 54 08 
00 69 9B 07 00 60 05 00 00 14 0D 00 00 0D 00 00 
00 20 00 00 00 FB FF FF FF 22 00 00 00 39 5A F9 
00 EE 00 00 00 D0 7E 6E 00 9B 00 00 00 DC CA 49 
32 00 00 00 00 00 00 00 00 B1 4F B5 62 01 22 14 
00 D0 94 8D 08 1A 17 0C 00 DF 01 00 00 05 00 00 
00 43 02 00 00 97 E6 ->B5 62 02 15<- 90 01 B6 F3 FD 
D4 44 84 01 41 66 08 12 0C 01 01 2E 17 2A 85 51 
62 E7 6A 77 41 A7 6B 2D F7 DB C3 9E 41 55 69 12 
45 00 14 00 00 F4 FB 26 04 03 07 07 00 E8 49 BF 
BA 1A 06 76 41 76 2B EC F4 1B EF 9C 41 BB 1D EF 

I cannot explain it myself and could not find any option that would solve the problem.

Software version:

• grabserial version 2.1.5
• Python 3.7.3
• Raspberry Pi 3 Model B Plus Rev 1.3
• GNU/Linux Raspbian GNU/Linux 10 (buster)

[AlphaSierraHotel]

The way that the --hex-output functionality deals with bytes from the serial port is different from how they are handled otherwise. For normal output, each single-byte is decoded using UTF-8 encoding with an option to ignore characters that it is unable to decode (it ignores/drops bytes with decimal values 128-255).

If you replace the word "ignore" in this line of code with the word, "backslashreplace" what do you get from the output of your first command line?

[AlphaSierraHotel]

Alternatively, try changing "utf-8" to "latin-1" in the same line of code.

[donok1]

Thanks @AlphaSierraHotel, this worked:

If you replace the word "ignore" in this line of code with the word, "backslashreplace"

After looking a bit more at the code, i noticed a few lines later that a condition is placed for writing in the output file. In : if out and out_char: I removed the condition on out_char, and it worked as well, raw data being written without any check: if out:

[AlphaSierraHotel]

Please try the alternative mentioned above. I believe it should then work without removing the check for out_char (because out_char will then be a valid character). If it works, we can possibly put forward a pull request to add an option.

[tbird20d]

Sorry to not look at this sooner. In reviewing the situation here, I think it would be good to have a 'raw' mode for grabserial, that just passes the bytes through from input (the serial port) to output (stdout and the log) with no unicode decoding or encoding at all. (I swear I've spent more time messing with the unicode issues with grabserial than any other aspect of the code.) The 'hexoutput' mode essentially omits the unicode handling, but also has changes the output. I think it would be relatively straightforward to create the new 'raw' mode, and have hexoutput automatically use that mode, and restrict the hexoutput flag to just controlling the output formatting. What do you think?

[AlphaSierraHotel]

I think we still need to do some decoding at least because we read a byte from the serial port and sometimes output it to stdout, a (string) text stream. I'd be happy to put together a pull request if you want. How would you feel about changing the default codec as a part of that?

[donok1]

I did a bit more troubleshooting (also fixed a RPi issue)

Please try the alternative mentioned above. I believe it should then work without removing the check for out_char (because out_char will then be a valid character). If it works, we can possibly put forward a pull request to add an option.

This worked. Using backslashreplace worked well, without the need to remove the validity check.

I also needed to remove the exclusion of the carriage return by commenting those 2 lines, otherwise it would just be omit them in the output file.

After checking the data against a recording using an SD card and OpenLog, it all seemed complete.

Finally, I'm all new here:

create the new 'raw' mode, and have hexoutput automatically use that mode, and restrict the hexoutput flag to just controlling the output formatting. What do you think?

but I think I might not be the only one to appreciate a raw mode :)

[donok1]

Once I got it running at full speed, I noticed that grabserial was not keeping up on my RaspberryPi Zero. The reading and writing of 1 single byte at the time is not the most efficient, and then sleeping for 1 second can already fill the buffer.

So I built myself a little script, with a rudimentary adaptative sleep time to save power usage. I have a buffer of 4098 bytes. But it would still be nice to enjoy some of the grabserial feature, such like the rotating filename and so on

with open(output_filename,"wb") as myFile:
    ser = setupSerial(GPSPort)
    ser.reset_input_buffer()
    ser.reset_output_buffer()

    Twait = 0.01    
    y1=0
    while True:
        y = ser.in_waiting
        if y > 50:
            z=ser.read(y)
            w = myFile.write(z) 
        else:
            if y1 < 200:
                Twait *= 1.3
            elif y1 > 500:
                Twait /= 2.5
            time.sleep(Twait)       
        y1 = y

If there would be a simple way to integrate some sort of intelligent use of the buffer, that would help a lot in performance.

Also, while looking through the pyserial doc, I noticed that flushOuput and flushInput used here where renamed with v3.0+. To be replaced by reset_output_buffer() and reset_input_buffer(). Sorry, I feel silly to do a PR just for that.

[github-actions[bot]]

Stale issue message