Simarine Pico Via invalid header

[skyweb07]

Hi @christopher-strack , thanks for this cool port from htool script, I wonder if you had time to test with other devices such as the Simarine Pico Via (for caravans).

I'm testing this script and for the moment as the header is hardcoded, it throws an invalid header error. Weardly is working with the Htool one where the header is dynamically calculated, is there any way to port that part so it's more generic? If you could guide me a little bit I can make the change as I have direct access to the caravan version of the pico and different devices installed.

let me know how I can help and I'll be glad to give support to this project, so maybe we can port it in the future and add it to Home Assistant, that would be awesome :)

[christopher-strack]

Hi @skyweb07,

I'm happy to help you, I'm definitely interested to make things work for other products. I made the parsing code stricter.

You can see the regular expression to parse the header here https://github.com/christopher-strack/spymarine/blob/main/spymarine/communication.py#L75. I'm happy to adapt the code if you could show the specific ParsingError you are getting and the bytes the Pico Via responds with.

As a quick fix, you can replace b"\x00\x00\x00\x00\x00\xff(.)\x85\xde\xc3\x46(..)\xff" with b"\x00\x00\x00\x00\x00\xff(.)....(..)\xff" I think.

[skyweb07]

this is what I'm getting after running the basic.py example code,

File "/opt/homebrew/lib/python3.11/site-packages/spymarine/communication.py", line 79, in parse_header
    raise ParsingError(f"Invalid header {data[:HEADER_LENGTH]!r}")
spymarine.error.ParsingError: Invalid header b'\x00\x00\x00\x00\x00\xff\x02\x7f\xfc\xbb\x1f\x00\x11\xff'
Exception ignored in: <function StreamWriter.__del__ at 0x105d19760>

the regular expression would not match it, I'm trying to extract same information with a generic regular expression but firstly need to find a common pattern that can't match for the moment

[christopher-strack]

Ok as expected the header parsing was a bit too strict. I fixed the issue in https://github.com/christopher-strack/spymarine/pull/12. Could you give that branch a try?

[skyweb07]

ok, now after a few tries, by changing the regular expression for b"\x00\x00\x00\x00\x00\xff(.)....(..)\xff" and changing the request_limit to 0.2 I was able to make it work, with 0.1 it was rejecting the connections, now I get almost all the sensors but one temperature sensor that's breaking. Also the data that is returning for most of the sensors doesn't seem valid, I'm trying to figure out why but still debugging, will open a PR with all the findings, thanks!

[skyweb07]

Ok as expected the header parsing was a bit too strict. I fixed the issue in #12. Could you give that branch a try?

that fixed the issue yes!, also needed to increase the request_limit to 0.2 or connections started getting rejected by pico

[christopher-strack]

with 0.1 it was rejecting the connections

I just tried this empirically for my device. It's a bit disappointing that the limit needs to be so high. Past firmware versions would work without any limit.

one temperature sensor that's breaking

Feel free to send me the bytes returned from your device and I can have a look as well

[skyweb07]

for example, here's the battery output data for my current battery bank, I've json formatted so it's easier to read

{
   "device_id":34,
   "name":"VAGABANK",
   "capacity":144.64,     =====> capacity is 800ah
   "battery_type":"lifepo4",
   "charge":{
      "id":3500,
      "value":-6.25e-05,   ======> percentage is 100%
      "unit":"percentage"
   },
   "remaining_capacity":{
      "id":3501,
      "value":-0.17,  =====> remaining is 800ah
      "unit":"ampere_hour"
   },
   "current":{
      "id":3502,
      "value":135.91, =====> this is something between 20-30ah
      "unit":"ampere"
   },
   "voltage":{
      "id":3503,
      "value":-1.056,  =====> 13.30v 
      "unit":"volt"
   },
   "type":"battery"
}

I'm testing with the different dict parameters to see if I can find the right values for those as seems invalid, then I can move to the other sensors

[christopher-strack]

@skyweb07 I see, that looks pretty far off. The parsing requires a complete understanding of all device types to know how many bytes to jump. I assume device_from_property_dict isn't ready yet for the Via or you have a product that is not yet parsed correctly like the Inclinometer.

If you could log the raw bytes received by Communication.request and post them here, I can try to add the missing pieces and add tests for the Via.

[skyweb07]

this would be the data response

first request:

b'\x00\x00\x00\x00\x00\xff\x02\x7f\xfc\xbb\x1f\x00\x11\xff\x01\x01\x00\x00\x003\xff\x02\x01\x00\x00\x008\xffa\xc4'

second request

b'\x00\x00\x00\x00\x00\xffA\x7f\xfc\xbb\x1f\x01a\xff\x00\x01\x00\x00\x00\x00\xff\x01\x03b_\xd4\xc1\xff\x00\x00\x00\n\xff\x02\x01\x00\x00\x00\x00\xff\x03\x01\x7f\xfc\xbb\x1f\xff\x04\x03e\xed\xe6\xf2\xff\x00\x00\x00\x01\xff\x05\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x06\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x07\x03d\xbc\t\xd0\xff\x00\x00\x00\x02\xff\x08\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\t\x03ef\x02\xc4\xff\x00\x00\x00\x01\xff\n\x04c\xf6E\xd2\xffSC269407\x00\xff\x0b\x03\x00\x00\x00\x00\xff\xc0\xa8\x01\x01\xff\x0c\x03\x00\x00\x00\x00\xff\x00\x00\x13\x89\xff\r\x03\x00\x00\x00\x00\xff\xc0\xa8\x01\xff\xff\x0e\x03\x00\x00\x00\x00\xff\x00\x00\xa8\xca\xff\x0f\x04c\xf6E\xd2\xffsc214726\x00\xff\x10\x03d\xbd\x14\x88\xff\x00\x00\x00\x01\xff\x11\x03e\x8e\x13\xd9\xff\x00\x00\x002\xff\x12\x03e\x8e\x13\xce\xff\x00\x00\x00\n\xff\x13\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x14\x03epeX\xff\x00\x00\x00\xb4\xff\x15\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x16\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x17\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x18\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x19\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x1a\x01\x00\x00\x00\x00\xff\x1b\x03\x00\x00\x00\x00\xff\x00\x00\x00<\xff\x1c\x03\x00\x00\x00\x00\xff\x00\x00\x00\x00\xff\x1d\x03d\xbc\x08\x96\xff\x00\x00\x00"\xffy\x83'

For the battery capacity just changing the property to use the raw number instead make the capacity correct at 800ah

elif device_type == 9:
        return Battery(
            device_id=device_id,
            name=property_dict.strings[3],
            capacity=property_dict.values[5].number / 100.0, <---- here
            battery_type=BatteryType(property_dict.values[8][1]),
        )`

I'll continue checking

[christopher-strack]

@skyweb07 Thanks but I would need the bytes of all requests to make sense of the situation

[skyweb07]

@skyweb07 Thanks but I would need the bytes of all requests to make sense of the situation

Here's the complete list of messages for 1 device request

b'\x00\x00\x00\x00\x00\xff\x02\x7f\xfc\xbb\x1f\x00\x11\xff\x01\x01\x00\x00\x003\xff\x02\x01\x00\x00\x008\xffa\xc4'

b'\x00\x00\x00\x00\x00\xffA\x7f\xfc\xbb\x1f\x01a\xff\x00\x01\x00\x00\x00\x00\xff\x01\x03b_\xd4\xc1\xff\x00\x00\x00\n\xff\x02\x01\x00\x00\x00\x00\xff\x03\x01\x7f\xfc\xbb\x1f\xff\x04\x03e\xed\xe6\xf2\xff\x00\x00\x00\x01\xff\x05\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x06\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x07\x03d\xbc\t\xd0\xff\x00\x00\x00\x02\xff\x08\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\t\x03ef\x02\xc4\xff\x00\x00\x00\x01\xff\n\x04c\xf6E\xd2\xffSC269407\x00\xff\x0b\x03\x00\x00\x00\x00\xff\xc0\xa8\x01\x01\xff\x0c\x03\x00\x00\x00\x00\xff\x00\x00\x13\x89\xff\r\x03\x00\x00\x00\x00\xff\xc0\xa8\x01\xff\xff\x0e\x03\x00\x00\x00\x00\xff\x00\x00\xa8\xca\xff\x0f\x04c\xf6E\xd2\xffsc214726\x00\xff\x10\x03d\xbd\x14\x88\xff\x00\x00\x00\x01\xff\x11\x03e\x8e\x13\xd9\xff\x00\x00\x002\xff\x12\x03e\x8e\x13\xce\xff\x00\x00\x00\n\xff\x13\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x14\x03epeX\xff\x00\x00\x00\xb4\xff\x15\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x16\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x17\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x18\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x19\x03\x00\x00\x00\x00\xff\x00\x00\x00\x01\xff\x1a\x01\x00\x00\x00\x00\xff\x1b\x03\x00\x00\x00\x00\xff\x00\x00\x00<\xff\x1c\x03\x00\x00\x00\x00\xff\x00\x00\x00\x00\xff\x1d\x03d\xbc\x08\x96\xff\x00\x00\x00"\xffy\x83'

b'\x00\x00\x00\x00\x00\xffA\x7f\xfc\xbb\x1f\x00T\xff\x00\x01\x00\x00\x00\x01\xff\x01\x03b_\xd4\xc1\xff\x00\x00\x00\x07\xff\x02\x01\x00\x00\x00\x00\xff\x03\x01\x7f\xfc\xbb\x1f\xff\x04\x03e\xee\xf2\x95\xff\x00\x00\x0e\x10\xff\x05\x03\x00\x00\x00\x00\xff\x00\x00\x00\x00\xff\x06\x03ep\xbe=\xff\x00\x00\x00\xff\xff\x07\x03\x00\x00\x00\x00\xff[\x81\xc4\x80\xffm\xfb'

b'\x00\x00\x00\x00\x00\xffA\x7f\xfc\xbb\x1f\x00l\xff\x00\x01\x00\x00\x00\x02\xff\x01\x03b_\xd4\xc1\xff\x00\x00\x00\x04\xff\x02\x01\x00\x00\x00\x00\xff\x03\x04\x00\x00\x00\x00\xffSensor 1\x00\xff\x04\x01\x7f\xfc\xbb\x1f\xff\x05\x01\x00\x00\x00\x01\xff\x06\x03\x00\x00\x00\x00\xff\x00\x00\x00\x00\xff\x07\x03\x00\x00\x00\x00\xff\x00\x00\x00\x00\xff\x08\x03ep\xbe=\xff\x00\x00\x00\xff\xff\t\x03\x00\x00\x00\x00\xffb1\xa5z\xff\xc1\x9e'

b'\x00\x00\x00\x00\x00\xffA\x7f\xfc\xbb\x1f\x00\x16\xff\x00\x01\x00\x00\x00\x03\xff\x01\x03b_\xd4\xc1\xff\x00\x00\x00\x00\xff\t\x04'

b'\x00\x01\x00\x00\x00\x04\xff\x01\x03\x00\x00\x00\x00\xff\x00\x00\x00\x00\xff'

[christopher-strack]

@skyweb07 The data still looks incomplete. The device count says you have 52 devices and the last message doesn't start with a valid header. Is that all data you get? Do you see an error after receiving those messages?

[skyweb07]

@skyweb07 The data still looks incomplete. The device count says you have 52 devices and the last message doesn't start with a valid header. Is that all data you get? Do you see an error after receiving those messages?

after that It repeat itself with the new data, i don't have 52 devices, but it may have those connections available as there are a lot of inputs available between the different devices i have (caravan panel, sp control panel for caravan with like 20 inputs, shunt with some inputs, dc-meter with 8 inputs, and some others, so it may be correct)

[christopher-strack]

@skyweb07 sorry I wasn't clear, I meant that you have 52 virtual devices, not real ones :) The first request just replies with the number of virtual devices. Afterwards the library sends a request for each device and the device replies with information about it.

In total, you should see 53 replies (or a Python error and one point). We would need to first make sure to extract all relevant information from your device or learn more about the errors.