InsufficientMemoryError occurred during unpack operation

[viandox]

• hoymiles_modbus version: 0.6.0
• Python version: 3.10
• Operating System: ubuntu focal

Description

Hi, first of all thanks for this work.

Finaly decided to open this issue after struggling hours with the following issue I initialy encountered when trying to use hoymiles-mqtt side project, with my hoymiles DTU-PRO:

Traceback (most recent call last):
  File "/home/yof/dtu/.venv/lib/python3.10/site-packages/plum/data.py", line 209, in unpack
    item, offset = cls.__unpack__(buffer, 0, None)
  File "<string>", line 254, in __unpack__
  File "/home/yof/dtu/.venv/lib/python3.10/site-packages/plum/int.py", line 201, in __unpack__
    raise InsufficientMemoryError("too few bytes to unpack")
plum.exceptions.InsufficientMemoryError: too few bytes to unpack

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yof/dtu/.venv/lib/python3.10/site-packages/plum/data.py", line 228, in unpack_and_dump
    item, offset = cls.__unpack__(buffer, 0, dmp.add_record(fmt=cls.name))
  File "<string>", line 288, in __unpack__
  File "/home/yof/dtu/.venv/lib/python3.10/site-packages/plum/int.py", line 205, in __unpack__
    chunk, end = getbytes(buffer, offset, dump, nbytes)
  File "/home/yof/dtu/.venv/lib/python3.10/site-packages/plum/_getbytes.py", line 42, in getbytes
    raise InsufficientMemoryError(unpack_shortage)
plum.exceptions.InsufficientMemoryError: 1 too few bytes to unpack uint8, 1 needed, only 0 available

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/yof/dtu/test.py", line 3, in <module>
    plant_data = HoymilesModbusTCP('192.168.0.146').plant_data
  File "/home/yof/dtu/.venv/lib/python3.10/site-packages/hoymiles_modbus/client.py", line 136, in plant_data
    microinverter_data = self.microinverter_data
  File "/home/yof/dtu/.venv/lib/python3.10/site-packages/hoymiles_modbus/client.py", line 114, in microinverter_data
    microinverter_data = self._microinverter_data_struct.unpack(result.encode()[1:41])
  File "/home/yof/dtu/.venv/lib/python3.10/site-packages/plum/data.py", line 216, in unpack
    cls.unpack_and_dump(buffer)
  File "/home/yof/dtu/.venv/lib/python3.10/site-packages/plum/data.py", line 242, in unpack_and_dump
    raise exceptions.UnpackError(dmp, exc) from exc
plum.exceptions.UnpackError: 

+--------+-----------+----------------------+-------+---------------------------------------+
| Offset | Access    | Value                | Bytes | Format                                |
+--------+-----------+----------------------+-------+---------------------------------------+
|        |           |                      |       | MISeriesMicroinverterData (Structure) |
|        | data_type | <insufficient bytes> |       | uint8                                 |
+--------+-----------+----------------------+-------+---------------------------------------+

InsufficientMemoryError occurred during unpack operation:

1 too few bytes to unpack uint8, 1 needed, only 0 available

What I Did

Simply trying to run sample code from documentation:

from hoymiles_modbus.client import HoymilesModbusTCP

plant_data = HoymilesModbusTCP('192.168.0.146').plant_data
print(plant_data.today_production)

DTU is connected throught Ethernet, is IP (ICMP) reachable and port 502 open. Same code explicitely ouptuts error when bad IPAddress is provided:

pymodbus.exceptions.ConnectionException: Modbus Error: [Connection] Failed to connect

Error also here when running from hoymiles-mqtt docker or python 3.10 on raspbery.

Thanks a lor for you help with whats look cryptic modbus error. Regards

[wasilukm]

Thank you for reporting this. Have you tried power cycling your DTU? Probably yes, just making sure. Could you please enable logging in your script? Add the following code at the beginning:

import logging
import pymodbus
pymodbus.pymodbus_apply_logging_config(logging.DEBUG)

[ttlappalainen]

Thanks for code - waiting to get it working.

I got same error. Log attached.

Log.txt

[ttlappalainen]

Just a note... I do not yet have panels and inverters running. Still snow on the roof. So there is no value for today production, but I expect it should return 0 not error.

[viandox]

Unfortunately for this issue, I managed to get DTU returned and refunded. I replaced it with cheap OpenDTU. I won't be able to proceed requested network capture and logging. I do confirm the DTU has been power cycled. Anyway thanks for your work!

[ttlappalainen]

It seems that DTU sends invalid response. E.g., request sno returns data length properly 3, Unit ID 0, function 3. Then there is 0 bytes data coming, while 4 registers (8 bytes) has been requested. Does not matter, what command or adress I try result is always same - 0 bytes data.

[wasilukm]

@ttlappalainen Indeed there should be data included in the response but it is not. There is nothing wrong with the library, and the only thing I can do is to improve error handling (make the exception more informative).

What DTU version do you have (HW and SW)? Is this DTU Pro-S? This issue is different from the one observed by others when DTU doesn't respond at all.

If you have extended access (installator rights) to your DTU then you may try playing with modbus setting, refer to https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwi514L9mKr9AhWkmIsKHS8IAloQFnoECCcQAQ&url=https%3A%2F%2Fwww.shinetech-power.de%2Fwp-content%2Fuploads%2F2021%2F07%2FTechnical-Note-Modbus-implementation-using-3Gen-DTU-Pro-V1.2.pdf&usg=AOvVaw1epboI0-XprGZWnV9mKnMd

Please also try reading other registers, perhaps your panels are in later registers. Each panel/port takes 40 registers, so add 40 (0x28) offset: 0x1001 + 0x28 + 0x28 + 0x28 + ... The total number of possible ports is 99.

[ttlappalainen]

Thanks for response.

Definetely problem is not in script, sorry I did not point that on my last mail. If you improve script error code, it could be something like: "device responded empty data". In DTU respond it partially right. As I pointed only problem is that it responds 0 length data, while 8 shold have been returned. On the other hand I did not find good exception code from modbus definitions for this case - "request is right, but for now I do not want to give any data". Maybe execption code 4 "An unrecoverable error occurred while the slave was attempting to perform the requested action" would be nearest.

As I mentioned I have device on table just for developing code. No inverters installed yet. Not sure could DTU be so stupid and then prevent all data. I would expect serial number would be returned in any case. But I do get same response from any address.

Device id DTU-Pro-S, Software Ver.V00.02.15, Hardware Ver.H09.06.01. I do not installer rights. I'll try to contact importer. Also will try with modbas RTU.

[wasilukm]

@ttlappalainen are your inverters added to DTU? Sorry for stupid question but the last comment from you confused me. Inverters have to be added by instalator to the system. DTU must be connected with hoymiles cloud to download the configuration.

[ttlappalainen]

Not stupid question. DTU and inverters has been added to cloud. But inverters are not yet connected at all - no solar or grid. I just have DTU on table powered and connected to network.

But sorry my mistake - I used wrong adress. So reading works for DTU sno at address 2000 and port number at 2501. I tried accidentaly read inverter sno, which doe not exist yet.

So since this is specified to Holymiles, it could instead of causing exception inform that "data is not available" or "inverter not available". DTU returns 0 length response, when trying to read from address, where is no inverter.

[ttlappalainen]

Or e.g., it could return just 0 for data. Then it would be possible to read sum on inverters total power without causing exception.

[wasilukm]

OK, so I understand that your inverters are not mapped to beginning registers and that's why the library doesn't find them.

For simplicity, the current implementation assumes that all inverters are mapped to consecutive registers starting from 0x1001, gaps are not supported. I can change it to support the whole range of registers no matter which ones are really occupied.

[ttlappalainen]

Now script started to work. Just tested again and now DTU returns data for inverters and shows U,I and P as 0. Also serial numbers will be shown for inverters. The difference is that now when device has been on long time, it shows on the cloud online. Before it showed offline even it was on already several hours. So somehow mapping took long time. But still strange is that before it did not repond with 0 values for inverters/panels. Now it shows 16 panels and 17th return 0 values.

[ttlappalainen]

More Chinese logic...

• Last data of 16 panels starts at address 1258. Returns OK data type 0x0C and sno OK.
• First missing slot starts at 1280. Returns OK data type 0x0C and sno 0.
• Next missing slot starts at 12A8. Returns OK data type 0x00 and sno 0.
• Next missing slot starts at 12D0. Returns double byte count as requested (which causes error) and data type 0x0C and sno 0.
• Seem to behave as above for further addresses.

Conclusion:

• Python script and QModMaster (Modbas tester) seem to expect that for requested register count related bytes should be returned. So if request is fo 4 registers and get 8 bytes, message is OK. DTU seem to return properly count of bytes in response, but when that does not match to requested count, other systems throws error.
• When DTU has not yet mapped any inverters, response from first inverter is 0 length.
• When there are mapped inverters, first empty slot response is OK with 0 sno.

Fix:

• Loop could be stopped to first 0 sno or 0 length response.
• For DTU response is OK, if is logically right. Meaning that there is as many bytes returned as response says even it does not match to requested count.
• Response for 0 inverters could cause execption with information "no inverters mapped yet" to avoid future issues.
• In principle unnecessary to do anything. Script works, when inverter has been properly mapped to DTU.

[wasilukm]

I'm happy that you have got it working. It seems that you confirmed that the library also works with DTU Pro-S Yes, Modbus implementation has some quirks (not to mention stability issues).