Checksum understanding

[mano8]

Hi karioja, and thank's for this good job. My english is litle bad sorry. Can you please explain me, checksum, i don't understand how it's work. In pdf victron veDirect protocol it say :

The statistics are grouped in blocks with a checksum appended. The last field in a block will always be “Checksum”. The value is a single byte, and will not necessarily be a printable ASCII character. The modulo 256 sum of all bytes in a block will equal 0 if there were no transmission errors. Multiple blocks are sent containing different fields.

But in the vedirect.py code bytes_sum global variable not contain checksum value, only 'Checksum' key and tabulation separator. But all seem working in my case and i don't understand how.

I made little changes in my fork. And use those changes but at this time i only tested with your vedirectsim script.

thank's a lot.

[karioja]

The bytes_sum variable is where we store the sum of each incoming byte. So whenever we receive a byte through the serial port it gets added to the bytes_sum variable. After we have received the whole packet we have 1) a sum of all the individual bytes in the packet and 2) checksum (last byte in the packet). After that we take modulo 256 of the sum and compare that to the checksum we received. If they match we have a packet that was uncorrupted in transit.

[mano8]

but here checksum not be compared no? What i don't understand is how sum of bytes can be ever 0 with modulo of 256 for variables input values. In your vedirectsim code when modifing values of inputs have lots of errors. In vedirect code sum of bytes modulo 256 have to be equal at 0. But for that, sum of bytes must be added with checksum value no?

[karioja]

Ah yes I made a mistake in my explanation above. It's been 2 years since I worker with this code. if (self.bytes_sum % 256 == 0): You are right. The sum and the checksum are not compared. Instead the checksum is added to the sum so that the end result is bytes_sum mod 256 equals 0. In the simulator it is done by this line result.append(256 - (sum(result) % 256)) So we calculate the sum of the packet, then take mod 256 of the sum and finally subtract that from 256 in order to get what needs to added as our checksum.

[mano8]

But in vedirect code checksum value not added at bit sum... Bit sum is stopped at checksum key no?

[mano8]

Then to checksum calculate value in vedirectsim i think result.append(256 - (sum(result+ord('\r')+ord('\n')) % 256)) result.append(ord('\r')) result.append(ord('\n')) because you add header1 and header2 at the top and at the end of bit sum no?

[mano8]

i do corective in in my vedirect fork and tested i think is right. Next for vedirectsim is to cut sending blocks width 18 blocks max (see in vedirect doc). In that case must have 2 send for complete data.

[karioja]

The VE.Direct packet is designed in a way where the checksum is the last byte in the packet. And a packet always begins with \r\n. So if you're trying to read the packet with something like python's readline or Arduino readBytesUntil it will fail.

[mano8]

Sorry for that it's right.

[karioja]

Yes the simulator does not enforce the 18 block limit. What I did originally was I combined all possible parameters from both MPPT charge controller and BVM battery monitor into a single python dict just to test that the protocol parsing works. It think I have a version of vedirectsim.py lying somewhere where I've actually separated those into two different dicts mpptdict and bvmdict and allow passing a parameter from command line to choose which parameters are sent by the simulator to allow simulating either BVM or MPPT data.

[mano8]

for clarify i thing have problem here:

elif self.state == self.IN_KEY: self.bytes_sum += ord(byte) if byte == self.delimiter: if (self.key == 'Checksum'): self.state = self.IN_CHECKSUM else: self.state = self.IN_VALUE else: self.key += byte return None

when checksum key detected, pass directly to checksum verrification and checksum value not be getted and added to byte_sum. No?

[mano8]

For 18 blocks limits i thing for the vedirect function : def read_data_single(self): In that case must have to do more for get all data:

• the first is bad data (depending time script start get only part of block)
• then first and second part of all data

[karioja]

The code is essentially a state machine and as such pretty hard to read.

if byte == self.delimiter: if (self.key == 'Checksum'): self.state = self.IN_CHECKSUM

So if the last byte that we read was \t and the last key that was read was 'Checksum' then move to checksum state. Checksum\t|WE ARE HERE|

We don't pass to the checksum validation directly. We only change the state of the state machine. Only when the next byte arrives we know that it's going to be the checksum and we can execute the checksum validation.

[mano8]

sorry it's possible i don't understand.. self.state control what whe have to get at this time no? For getting checksum value state must : self.state = self.IN_VALUE at end of Checksum\t|WE ARE HERE| No?

And then when checksum value getted go to self.state = self.IN_CHECKSUM.

sorry if i do not understand

[karioja]

For 18 blocks limits i thing for the vedirect function : def read_data_single(self): In that case must have to do more for get all data:

* the first is bad data (depending time script start get only part of block)

* then first and second part of all data

read_data_single will only read a single valid packet from the serial stream and does not care whether it's the full data or not. vedirect.py tries to be a protocol decoder and the interpretation whether we have the full data set or not has to be done in the layer above. From what I gather from the specification is that the 18 block limit applies to BVM only. MPPT has exactly 18 parameters while BVM has more than that. My MPPT (fw. 1.19) send the same 18 parameters in each packet.

[karioja]

sorry it's possible i don't understand.. self.state control what whe have to get at this time no? For getting checksum value state must : self.state = self.IN_VALUE at end of Checksum\t|WE ARE HERE| No?

And then when checksum value getted go to self.state = self.IN_CHECKSUM.

sorry if i do not understand

self.state controls what happens in the next round when input is called again with a new byte from the serial stream.

You can think of IN_CHECKSUM state as equivalent to IN_VALUE state where we read the value but in addition to that it marks the end of the packet where we do the checksum validation and reset the state machine and return the valid packet.

[mano8]

thank's very much for your help and your time.

Checksum\t|WE ARE HERE| is then end of the packet? Here the final of code process (i put print in some parts of code):

• header1 getted
• header2 getted
• input key getted
• value getted
• header2 getted
• Checksum key getted
• checksum validation
• Packet validated
• return Packet In this case value of checksum is not getted and added to byte_sum. Must be :
• ........
• Checksum key getted - value Checksum getted
• checksum validation
• Packet validated
• return Packet

End of packet must be Checksum\tChecksumValue|WE ARE HERE| No?

[karioja]

End of packet must be Checksum\tChecksumValue|WE ARE HERE| No?

Correct.

[mano8]

Sorry i understanded my mystake thank's for all. you add checksum value at end. what i do is multibit checksum. The errors i encountered in vedirectsim when i modify input values seem to be when checksum value is 256. here data to test : self.dict = {'V': '12800', 'VS': '12800', 'VM': '1280', 'DM': '120', 'VPV': '3350', 'PPV': '130', 'I': '-15000', 'IL': '1500', 'LOAD': 'ON', 'T': '25', 'P': '130', 'CE': '13500', 'SOC': '876', 'TTG': '45', 'Alarm': 'OFF', 'Relay': 'OFF', 'AR': '1', 'H1': '55000', 'H2': '15000', 'H3': '13000', 'H4': '230', 'H5': '12', 'H6': '234000', 'H7': '11000', 'H8': '14800', 'H9': '7200', 'H10': '45', 'H11': '5', 'H12': '0', 'H13': '50', 'H14': '0', 'H15': '11500', 'H16': '14800', 'H17': '34', 'H18': '45', 'H19': '456', 'H20': '45', 'H21': '300', 'H22': '45', 'H23': '350', 'ERR': '0', 'CS': '5', 'BMV': '702', 'FW': '1.19', 'PID': '0x204', 'SER#': 'HQ141112345', 'HSDS': '0'} sorry for all.

[karioja]

Yes, you seem to have found a problem with the checksum generation in vedirectsim.py. Fortunately the fix is easy. result.append((256 - (sum(result) % 256)) % 256)

[mano8]

thank you very much for all sincerely