Control characters in output

[pml-jpe]

Two of the things I am logging have strange character showing in the output. Here is an example of one:

I think these are RS232 control characters. Interestingly these do not show when I capture the serial using Tera Term:

I don't want to see or log these characters. Is there anything I can do?

[janipewter]

Anyone?

[tbird20d]

Sorry - I missed this earlier. grabserial tries to preserve data as best it can from the serial port. It does not have any character-level filtering option. I primarily use Linux, and on Linux this would be trivial to strip from standard out using 'tr' or from the log file using 'sed'. This is somewhat more difficult in Windows, but you could look at this https://stackoverflow.com/questions/29291986/what-is-the-tr-command-in-windows and see if it does what you want.

[janipewter]

Thank you for the reply. I agree it is best used on Linux, but unfortunately in this specific use case I have to use Windows 10. I have looked into the tr option and it seems all avenues are too hack-y for my liking. Regardless I want to use grabserial's built-in logging rather than messing around with sed etc after the fact.

I think the underlying issue is caused perhaps due to each line of data having a checksum. I am not proficient enough in Python to fully understand the code, but is there a way I could modify the grabserial code to not print the control characters? I don't need this added to the public version; I am happy to keep a modified version locally.

Thanks again

[tbird20d]

OK - no problem. I wrote a very simple (and very specific to your use-case) fix, and put this on the branch "skip_nonprintable". The commit is here: https://github.com/tbird20d/grabserial/commit/e2cfafe8756fed5193626b9dd907cddda07afa02 If you see other control characters in your output that you want to skip, you should be able to add them to that if statement to also skip those. Please give it a try and let me know if it works for you.

[janipewter]

Thank you. I switched to that branch and gave it a try and the characters are still printed and logged. Do you have any suggestions of what else I could try? No need to push commits, I am happy to make changes locally for testing.

[tbird20d]

Well that's odd. Can you attach a log? I'd like to look at the contents and see if I can figure out what's going on. Thanks.

[janipewter]

202103162100-Wind.txt

Here you go.

[tbird20d]

I looked at a hexdump of the log: 00000000 5b 54 75 65 20 4d 61 72 20 31 36 20 32 31 3a 34 |[Tue Mar 16 21:4| 00000010 38 3a 33 30 2e 34 36 32 33 38 38 20 32 30 32 31 |8:30.462388 2021| 00000020 5d 20 4d 2c 2b 30 31 34 2e 33 37 2c 30 30 2c 03 |] M,+014.37,00,.| 00000030 30 46 0a 5b 54 75 65 20 4d 61 72 20 31 36 20 32 |0F.[Tue Mar 16 2| 00000040 31 3a 34 38 3a 33 30 2e 35 34 32 39 33 38 20 32 |1:48:30.542938 2| 00000050 30 32 31 5d 20 02 51 2c 2d 30 30 31 2e 35 33 34 |021] .Q,-001.534| 00000060 2c 2b 30 30 30 2e 32 39 30 2c 2b 30 30 30 2e 35 |,+000.290,+000.5| 00000070 30 39 2c 4d 2c 2b 30 31 34 2e 33 37 2c 30 30 2c |09,M,+014.37,00,| 00000080 03 30 33 0a 5b 54 75 65 20 4d 61 72 20 31 36 20 |.03.[Tue Mar 16 |

and the codes are clearly ASCII 02s and 03s, so the filtering code I provided should work.

I wrote a test program using the same logic, to output

the log with the codes stripped, and it worked. Here is the test program:

!/usr/bin/python

import sys

fd = open("sl.txt", "r") while True: x = fd.read(1) if not x: break if x == "\002" or x == "\003": continue sys.stdout.write(x) sys.stdout.flush()

fd.close()

---

I'm at a loss for why the grabserial code failed to strip the codes. Can you please check and make sure you were running the version of grabserial from the strip_nonprintable branch?

Maybe add the following to debug the code:

Where the STX and ETX check are, add some extra prints:

        # if we didn't read anything, loop
        if len(x) == 0:
            continue

        print(" %02X" % ord(x))

        # ignore STX and ETX control characters
        if x == '\002' or x == '\003':
            print("skipping one")
            continue

Let me know what you see.

[janipewter]

I'm definitely on the right branch, I can see the extra code in the script. When I ran it with the debug prints in, this happened:

Interestingly the logging was unaffected (new logfile attached). I commented out the print(" %02X" % ord(x)) line which unbroke the printed output. The control characters are still in stdout and the logfile.

202103171800-Wind.txt

[tbird20d]

OK - the only thing I can think of is that maybe the literals are not working in Windows the way they work in Linux.

Can you try this instead:

Replace:

ignore STX and ETX control characters

        if x == '\002' or x == '\003':
            continue

with

ignore STX and ETX control characters

        if ord(x) == 2 or ord(x) == 3:
            continue

[pml-jpe]

That works perfectly. Thank you so much. Could you please see my other issue too: https://github.com/tbird20d/grabserial/issues/55 (this one is less critical)

[github-actions[bot]]

Stale issue message