Read bytes directly from a LoRa module over a Pi serial port

[SteveCossy]

Previous implementation of Cicadacom protocol has always been on a PicAxe processor. This usually requires two steps:

1. Read data from a sensor and transmit it over a physical wireless link (recently, LoRa).
2. Receive data as bytes on a compatible wireless link, convert it to ASCII, and send to a Pi via wired serial link.

This issue will remove the need for a PicAxe board in step 2.

The Cicadacom protocol is defined here: https://github.com/SteveCossy/IOT/wiki/CayenneDataChannels-PicAxeChannels-Cicadacom The resulting code is here: https://github.com/SteveCossy/IOT/tree/master/LoRaReAd

At the time of opening this issue, there are four lines of code that do some work:

while True:
   with serial.Serial('/dev/ttyS0', 2400, timeout=50) as ser:
      x = ser.read(100)
      print( x )

Typical output is:

b':01G)\x00_:01H\tJ::01I\xae\x00\xd6:01J\xce\xb7\x02:01K&\x00\\'
b':01L\xfc\x0e\x8f:01G)\x00_:01H\xf2I\xc2'
b':01I\xae\x00\xd6:01J\xb2\xb7~:01K&\x00\\:01G)\x00_:01H\xf2I\xc2'
b':01I\xae\x00\xd6:01J\xb2\xb7~:01K&\x00\\:01G)\x00_:01H\xf2I\xc2'
b':01I\xae\x00\xd6:01J\xb2\xb7~:01K&\x00\\:01G)\x00_:01H\xf2I\xc2'
b':01I\xae\x00\xd6:01J\xb2\xb7~:01K&\x00\\:01G)\x00_:01H\xf2I\xc2'
b':01I\xae\x00\xd6:01J\xb2\xb7~:01K&\x00\\:01G)\x00_'

Other research/results are documented here: https://docs.google.com/spreadsheets/d/1yAlynZ3Xm06lQpewKZHlqzAt-p85bqLf5aJdzJMf9Kk/edit#gid=0

[shermp]

A few thoughts from me.

I had discussions with Andrew a few weeks ago, and the idea was floated to make the picaxe channel letter a number instead, to match the cayenne channel number. It could help avoid any confusion.

I see in the proposed data structure there is a possible mix of int, word and byte. Is that 32, 16, 8 bits respectively? If this is correct, is 2 bytes enough to be sending? Or should another couple of bytes be added to bring the data size to 4 bytes? Then treat it as a single 4 byte big (or little) endian value at the receiving end?

Python has the struct module for packing/unpacking binary data to python types.

[SteveCossy]

I am not sure what the 'K' channel does - seems to be for random data. Perhaps this could be used for a data structure that requires 4 bytes - just send 16 of the bits in another packet. My understanding is that none of the current sensors use more than 16 bits. Int is, I think, either 8 or 16 bits. Yes - have started to experiment with the struct module (from my sheet): *>>> t4 = b'\x08J;\x00'

len(t4)
4
">>> struct.unpack(""f"",t4)"
(5.4448516818173654e-39,) Needs more thinking. 🤔

1. Get the right bytes (ones that are data).
2. Evaluate them in the right order. Can't be too hard!

[shermp]

The idea with the struct module is to do something like:

>>> struct.pack("<2schb", b":0", b"A", 3000, 0)
b':0A\xb8\x0b\x00'
>>> packed = struct.pack("<2schb", b":0", b"A", 3000, 0)
>>> head, chan, data, chk = struct.unpack("<2schb", packed)
>>> head
b':0'
>>> chan
b'A'
>>> data
3000
>>> chk
0
>>> 

Note the '<' character in the format string. That states that the data will be little endian, and most importantly, no padding bytes will be introduced or expected. For reading data from a network/file/serial, one almost always wants to prefix the format string with one of '>' '<' or '!'

[SteveCossy]

Looks like I have the Python code happily getting all the information it needs out of the Cicadacom data! Most the code not currently used will be needed to send the information to Cayenne. Will tidy out the rest. Thoughts welcome .... https://github.com/SteveCossy/IOT/blob/master/LoRaReAd/LoRa_to_MQTT.py

I can't see a way of processing the two bytes of data using struct.unpack, because that returns a signed long int, and I need it unsigned. I've put in some code to do this manually. Will leave it running overnight. Tomorrow I'll check that there are no failed checksums. Here is an example of output from the script:

python3 LoRa_to_MQTT.py
{'CayUsername': '6375a470-cff9-11e7-86d0-83752e057225', 'CayPassword': '26e1dc13f900da7b30b24cad4b320f9bc6dd0d78', 'CayClientID': 'f69ea390-f519-11e9-b49d-5f4b6757b1bf', 'UniqueID': 'PythonClient0xb827ebb46f3e'}
b':01J\xc9\xb7\x05' 7
b':'
b'0'
b'1'
b'J'
-18487
5
Calculated data:  47049
Checksum correct:  True
b':01K%\x00_' 7
b':'
b'0'
b'1'
b'K'
37
95
Calculated data:  37
Checksum correct:  True
b':01G)\x00_' 7
b':'
b'0'
b'1'
b'G'
41
95
Calculated data:  41
Checksum correct:  True
b':01H\x04J7' 7
b':'
b'0'
b'1'
b'H'
18948
55
Calculated data:  18948
Checksum correct:  True
b':01I\xae\x00\xd6' 7
b':'
b'0'
b'1'
b'I'
174
214
Calculated data:  174
Checksum correct:  True
b':01J\xc9\xb7\x05' 7
b':'
b'0'
b'1'
b'J'
-18487
5
Calculated data:  47049
Checksum correct:  True
b':01K%\x00_' 7
b':'
b'0'
b'1'
b'K'
37
95
Calculated data:  37
Checksum correct:  True
b':01H\x04J7' 7
b':'
b'0'
b'1'
b'H'
18948
55
Calculated data:  18948
Checksum correct:  True
b':01I\xae\x00\xd6' 7
b':'
b'0'
b'1'
b'I'
174
214
Calculated data:  174
Checksum correct:  True
b':01J\xc9\xb7\x05' 7
b':'
b'0'
b'1'
b'J'
-18487
5
Calculated data:  47049
Checksum correct:  True
b':01K%\x00_' 7
b':'
b'0'
b'1'
b'K'
37
95
Calculated data:  37
Checksum correct:  True
b':01G)\x00_' 7
b':'
b'0'
b'1'
b'G'
41
95
Calculated data:  41
Checksum correct:  True
b':01H\x04J7' 7
b':'
b'0'
b'1'
b'H'
18948
55
Calculated data:  18948
Checksum correct:  True
b':01I\xae\x00\xd6' 7
b':'
b'0'
b'1'
b'I'
174
214
Calculated data:  174
Checksum correct:  True
b':01J\xc9\xb7\x05' 7
b':'
b'0'
b'1'
b'J'
-18487
5
Calculated data:  47049
Checksum correct:  True
b':01K%\x00_' 7
b':'
b'0'
b'1'
b'K'
37
95
Calculated data:  37
Checksum correct:  True
b':01G)\x00_' 7
b':'
b'0'
b'1'
b'G'
41
95
Calculated data:  41
Checksum correct:  True
b':01H\x04J7' 7
b':'
b'0'
b'1'
b'H'
18948
55
Calculated data:  18948
Checksum correct:  True
b':01I\xae\x00\xd6' 7
b':'
b'0'
b'1'
b'I'
174
214
Calculated data:  174
Checksum correct:  True
b':01J\xc9\xb7\x05' 7
b':'
b'0'
b'1'
b'J'
-18487
5
Calculated data:  47049
Checksum correct:  True
b':01K%\x00_' 7
b':'
b'0'
b'1'
b'K'
37
95
Calculated data:  37
Checksum correct:  True
b':01G)\x00_' 7
b':'
b'0'
b'1'
b'G'
41
95
Calculated data:  41
Checksum correct:  True
b':01H\x04J7' 7
b':'
b'0'
b'1'
b'H'
18948
55
Calculated data:  18948
Checksum correct:  True

[shermp]

If you need unsigned numbers, just use the correct format character. For 16 bit numbers, that's 'H' instead of 'h'. Python will do the rest.

There is also no need to unpack the data a second time to calculate the checksum. Just index the bytes directly. They will be integers rather than bytes, but the xor should still work.

(cicadaPacket[2] ^ cicadaPacket[3] ^ cicadaPacket[4] ^ cicadaPacket[5]) == cicadaPacket[6] 

[SteveCossy]

Thanks again! Not sure how I missed the 'H' format! Shortened the Checksum down to:

      CksTest = 0
      for x in range(2,6):
          CksTest = CksTest ^ x

Then CksTest should equal 0! :-) This tests OK. Will implement later today.

[shermp]

Umm.. Not sure that will work. Your not iterating over bytes there, just a number. You need to use x as an index, or better yet, iterate like:

CksTest = 0
for byte in cicadaPacket[2:6]:
    CksTest = CksTest ^ byte

[SteveCossy]

Oops! Did have a variation working in Python command line, but copied the wrong test line into the code. I'm now using:

      CksTest = 0
      for byte in PacketIn[2:7]:
          CksTest = CksTest ^ byte
          print(byte, CksTest)

The way a for loop differs from other languages gets to me. Doesn't iterate with the final value, but exists the loop, so 2:7 iterates the loop 5 times, using 2,3,4,5,6 but not '7'. This time I put in some sensible debugging, to confirm it is actually doing something!

b':01G)\x00_' 7
49 49
71 118
41 95
0 95
95 0
b'1' b'G' 41 95
Checksum correct!
b':01H\x08J;' 7
49 49
72 121
8 113
74 59
59 0
b'1' b'H' 18952 59
Checksum correct!
b':01I\xae\x00\xd6' 7
49 49
73 120
174 214
0 214
214 0
b'1' b'I' 174 214
Checksum correct!
b':01J\xd0\xb7\x1c' 7
49 49
74 123
208 171
183 28
28 0
b'1' b'J' 47056 28
Checksum correct!