Record layout for T000104

[birgerj]

I have an older model ShineWiFiBox with data layout T000104. From what I've read there needs to be a recordlayout json present to that Grott can decode this. This seems to be done by reverse engineering the data. What can I do to reverse engineer this data? Is there any documentation on these layouts?

This is what I see in the log:

- Growatt packet received:
    <socket.socket fd=11, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('192.168.64.250', 50226), raddr=('47.91.67.66', 5279)>
- Growatt original Data:
    \x00\x01\x00\x00\x00\xd3\x01\x04\x41\x48\x33\x30\x38\x31\x31\x32\x38\x31\x42
    \x58\x33\x31\x35\x31\x31\x36\x31\x38\x02\x00\x00\x00\x2c\x00\x00\x00\x00\x00
    \x00\x0a\xe5\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
    \x00\x00\x13\x84\x09\x3b\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
    \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x45\xa6\x12\x4b
    \xf3\x6c\x00\xac\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
    \x00\x00\x00\x0a\xed\x00\x00\x00\x00\x00\x2d\x00\x59\x4e\x20\x00\x00\x00\x00
    \x00\x00\x00\x00\x00\x06\xab\x90\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\xab
    \x90\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
    \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
    \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
    \x00\x00\x00\x00\x00\x00\x00\x00
- Grott automatic protocol detection
- Grott data record length 217
- layout   :  T000104
- no matching record layout found, try generic
- no matching record layout found, standard processing performed
- Record layout used :  none
- Growatt data decrypted V2
- Grott Growatt data decrypted
- Growatt plain data:
    0001000000d30104063a5c47594545754a5e3539474572435e41504c7647726f5b61747447726
    f7d84747447726f7761747447726f7761747447727cf3684f7447726f7761747447726f776174
    7447726f7761747447726f77617231e16024840d74d847726f7761747447726f7761747447726
    f776b997447726f774c742d09526f7761747447726f7767dfe447726f77617474477269dcf174
    7447726f7761747447726f7761747447726f7761747447726f7761747447726f7761747447726
    f7761747447726f7761747447726f7761747447726f776174
- Grott data ack record or data record not defined no processing done

[Neflardio]

You will want to look at the full log. It looks like that particular section of the log is for one of the random growatt messages that doesn't actually have solar data.

[johanmeijer]

Oke 00 as " protocol" I have never seen before. Normally older installations have 02 (T020104).

The "protocol" describes the basic record layout (e.g is encryption used, where are the serial numbers of the inverter and datalogger located, which registers are in the record, where starts the datarecord etc).

It does not say anything about the data in the record. For that, we have to know the inverter type. What kind of inverter do you have?

You can try to fall back to the basic grott decoding by specifying these parameters in the grott.ini?

compat = False valueoffset = 6 decrypt = False

You might have to play a little bit with the valueoffset (it determines where the data record starts).

In the meantime, I will try to decode the record, but inverter type is important for that.

The 04 means it is a data record and should contain solar data. But because grott tries to decrypt an uncrypted record, it is not readdable anymore. The original data is the uncrypted data we have to use to find out the layout.

[birgerj]

Thanks for picking this up @johanmeijer!

I made the changes to the ini file and now this is the output (see below).

The inverter is model Growatt GT0G000F141. Sadly I couldn't find much about this model besides this manual: https://docplayer.nl/44213376-Handleiding-voor-installatie-gebruik-monofasige-growatt-omvormers.html It was already installed here when I bought the house, no other documentation available here.

I do get this out of the messages that are received so there some data coming in. Every line is a new message (received with 1 minute intervals)

SSID: AH30811281, Channel: 1, Power Level: 32, Model: AH30811281, Zone: 2, IP Address: 192.168.10.100, Port: 3252.
SSID: AH30811281, MAC Address: AC:CF:23:0F:A2:14, IP Address: 192.168.64.250, Port: 5279, Firmware Version: 1.0.0.0
SSID: AH30811281B, Serial Number: X31511618, Model: Growatt Inverter0G29509B
SSID: AH30811281B, Serial Number: X31511618, Custom Field: E.
SSID: AH30811281B, Serial Number: X31511618
SSID: AH30811281, MAC Address: AC:CF:23:0F:A2:14, IP Address: 192.168.64.250, Port: 5279, Firmware Version: 1.0.0.0
SSID: AH30811281B, Serial Number: X31511618, Custom Field: M, Operating Mode: YN.

also there are many messages like:

Growatt packet received:
        <socket.socket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('192.168.64.250', 59557), raddr=('47.91.67.66', 5279)>
    - Data less then minimum record length, data not processed

Can it be that these contain some useful info but are discarded because they seem too small?

This is an example of a message that does get received and which I decoded above:

- Growatt packet received:
    <socket.socket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('192.168.64.250', 59557), raddr=('47.91.67.66', 5279)>
- Growatt original Data:
    \x00\x01\x00\x00\x00\x11\x01\x19\x41\x48\x33\x30\x38\x31\x31\x32\x38\x31\x00
    \x05\x00\x01\x31\x00\x01\x00\x00\x00\x12\x01\x19\x41\x48\x33\x30\x38\x31\x31
    \x32\x38\x31\x00\x06\x00\x02\x33\x32\x00\x01\x00\x00\x00\x1a\x01\x19\x41\x48
    \x33\x30\x38\x31\x31\x32\x38\x31\x00\x08\x00\x0a\x41\x48\x33\x30\x38\x31\x31
    \x32\x38\x31\x00\x01\x00\x00\x00\x11\x01\x19\x41\x48\x33\x30\x38\x31\x31\x32
    \x38\x31\x00\x0d\x00\x01\x32\x00\x01\x00\x00\x00\x1e\x01\x19\x41\x48\x33\x30
    \x38\x31\x31\x32\x38\x31\x00\x0e\x00\x0e\x31\x39\x32\x2e\x31\x36\x38\x2e\x31
    \x30\x2e\x31\x30\x30\x00\x01\x00\x00\x00\x14\x01\x19\x41\x48\x33\x30\x38\x31
    \x31\x32\x38\x31\x00\x0f\x00\x04\x33\x32\x35\x32
- Grott automatic protocol detection
- Grott data record length 164
- layout   :  T000119
- no matching record layout found, try generic
- Record layout used :  T000119
- Growatt data decrypted V2
- Grott Growatt data decrypted
- Growatt plain data:
    0001000000110119063a5c47594545754a5e7764747576726e77617466466b2e3f52444c76435
    d4f50747247705c456175744772757678353c7442574650464c767267776b353c744257465046
    4c76726e77617465466b2e3f52444c76435d4f50747947735d77607474476c6e6e203c47774a5
    e46534c45477c6f79504d466943594f4f454469435f4761757447727b7678353c744257465046
    4c767260776547467240
- Grott data ack record or data record not defined no processing done
- Growatt packet received:
    <socket.socket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('192.168.64.250', 59557), raddr=('47.91.67.66', 5279)>
- Growatt original Data:
    \x00\x01\x00\x00\x00\x21\x01\x19\x41\x48\x33\x30\x38\x31\x31\x32\x38\x31\x00
    \x10\x00\x11\x41\x43\x3a\x43\x46\x3a\x32\x33\x3a\x30\x46\x3a\x41\x32\x3a\x31
    \x34\x00\x01\x00\x00\x00\x1e\x01\x19\x41\x48\x33\x30\x38\x31\x31\x32\x38\x31
    \x00\x11\x00\x0e\x31\x39\x32\x2e\x31\x36\x38\x2e\x36\x34\x2e\x32\x35\x30\x00
    \x01\x00\x00\x00\x14\x01\x19\x41\x48\x33\x30\x38\x31\x31\x32\x38\x31\x00\x12
    \x00\x04\x35\x32\x37\x39\x00\x01\x00\x00\x00\x1e\x01\x19\x41\x48\x33\x30\x38
    \x31\x31\x32\x38\x31\x00\x13\x00\x0e\x31\x39\x32\x2e\x31\x36\x38\x2e\x36\x34
    \x2e\x32\x35\x30\x00\x01\x00\x00\x00\x17\x01\x19\x41\x48\x33\x30\x38\x31\x31
    \x32\x38\x31\x00\x15\x00\x07\x31\x2e\x30\x2e\x30\x2e\x30
- Grott automatic protocol detection
- Grott data record length 166
- layout   :  T000119
- no matching record layout found, try generic
- Record layout used :  T000119
- Growatt data decrypted V2
- Grott Growatt data decrypted
- Growatt plain data:
    0001000000210119063a5c47594545754a5e7771746506315534274e4674485f315b35467d435
    b77607474476c6e6e203c47774a5e46534c4547636f79504d466943594f4f424069405a476175
    7447727b7678353c7442574650464c76727d77654146704b6f76617474597376362947447f435
    e455945745472614658465a7644575957405a75475f7760747447656e6e203c47774a5e46534c
    4547676f70505a4469424147
- Grott data ack record or data record not defined no processing done

[johanmeijer]

I create a layout file. Can you test these one for me?

You can find it in the examples directory (of the beta branch):

https://github.com/johanmeijer/grott/tree/Beta-(2.8.x)/examples/Record%20Layout

Please copy t0000104,json in your grott home directory at your server and restart Grott.

Be aware the grott.ini settings as described above should be reset to the default values (or be deleted).

[birgerj]

I've downloaded t000104.json to the /app folder. I thought there was a case-issue so also copied to T000104.json:

-rw-r--r-- 1 root root 34541 Oct 20 12:55 T000104.json
drwxr-xr-x 1 root root   200 Oct 13 15:01 __pycache__
-rw-rw-r-- 1 root root  2845 Dec 14  2021 grott.ini
-rw-rw-r-- 1 root root  1449 Dec 14  2021 grott.py
-rw-rw-r-- 1 root root 84994 Dec 14  2021 grottconf.py
-rw-rw-r-- 1 root root 27695 Dec 14  2021 grottdata.py
-rw-rw-r-- 1 root root  6792 Dec 14  2021 grottproxy.py
-rw-rw-r-- 1 root root  5468 Dec 14  2021 grottsniffer.py
-rw-r--r-- 1 root root 34541 Oct 20 12:46 t000104.json

And during start up:

Grott process json layout files
t000104.json
T000104.json

So I think the layout files are discovered.

But when processing incoming message:

Grott automatic protocol detection
- Grott data record length 217
- layout   :  T000104
- no matching record layout found, try generic
- no matching record layout found, standard processing performed
- Record layout used :  none
- Growatt data decrypted V2
- Grott Growatt data decrypted

Which seems to me the file layout file isn't picked up somehow? I did I miss anything? I've reset the grott.ini file to the default values.

Note I'm running docker grott:latest. Should I be running grott:beta?

[johanmeijer]

Ok. It works in my (windows) test environment (with only the t000104.json). Can you delete the T000104.json? maybe it can not handles both files? I have to look at Linux (that understand the difference between a t and a T) if that is the case. But only production is running there now.

[johanmeijer]

You can see if the layout is loaded just before the: Grott external record whitelist 'recwl.txt' not found message. It looks like:

T000104 : {'decrypt': {'value': 'False'}, 'pvserial': {'value': 36, 'length': 10, 'type': 'text', 'divide': 10}, 'date': {'value': 56, 'divide': 10}, 'regstart1': {'value': 58, 'length': 2, 'type': 'num', 'incl': 'no'}, 'regend1': {'value': 62, 'length': 2, 'type': 'num', 'incl': 'no'}, 'pvstatus': {'value': 66, 'length': 2, 'type': 'num'}, 'pvpowerin': {'value': 70, 'length': 4, 'type': 'num', 'divide': 10}, 'pv1voltage': {'value': 78, 'length': 2, 'type': 'num', 'divide': 10}, 'pv1current': {'value': 82, 'length': 2, 'type': 'num', 'divide': 10}, 'pv1watt': {'value': 86, 'length': 4, 'type': 'num', 'divide': 10}, 'pv2voltage': {'value': 94, 'length': 2, 'type': 'num', 'divide': 10}, 'pv2current': {'value': 98, 'length': 2, 'type': 'num', 'divide': 10}, 'pv2watt': {'value': 102, 'length': 4, 'type': 'num', 'divide': 10}, 'pvpowerout': {'value': 110, 'length': 4, 'type': 'num', 'divide': 10}, 'pvfrequentie': {'value': 118, 'length': 2, 'type': 'num', 'divide': 100}, 'pvgridvoltage': {'value': 122, 'length': 2, 'type': 'num', 'divide': 10}, 'pvgridcurrent': {'value': 126, 'length': 2, 'type': 'num', 'divide': 10}, 'pvgridpower': {'value': 130, 'length': 4, 'type': 'num', 'divide': 10}, 'pvgridvoltage2': {'value': 138, 'length': 2, 'type': 'num', 'divide': 10}, 'pvgridcurrent2': {'value': 142, 'length': 2, 'type': 'num', 'divide': 10}, 'pvgridpower2': {'value': 146, 'length': 4, 'type': 'num', 'divide': 10}, 'pvgridvoltage3': {'value': 154, 'length': 2, 'type': 'num', 'divide': 10}, 'pvgridcurrent3': {'value': 158, 'length': 2, 'type': 'num', 'divide': 10}, 'pvgridpower3': {'value': 162, 'length': 4, 'type': 'num', 'divide': 10}, 'pvenergytoday': {'value': 170, 'length': 4, 'type': 'num', 'divide': 10}, 'pvenergytotal': {'value': 178, 'length': 4, 'type': 'num', 'divide': 10}, 'totworktime': {'value': 186, 'length': 4, 'type': 'num', 'divide': 7200}, 'pvtemperature': {'value': 194, 'length': 2, 'type': 'num', 'divide': 10}, 'isof': {'value': 198, 'length': 2, 'type': 'num', 'divide': 1}, 'gfcif': {'value': 202, 'length': 2, 'type': 'num', 'divide': 1}, 'dcif': {'value': 206, 'length': 2, 'type': 'num', 'divide': 1}, 'vpvfault': {'value': 210, 'length': 2, 'type': 'num', 'divide': 1}, 'vacfault': {'value': 214, 'length': 2, 'type': 'num', 'divide': 1}, 'facfault': {'value': 218, 'length': 2, 'type': 'num', 'divide': 1}, 'tmpfault': {'value': 222, 'length': 2, 'type': 'num', 'divide': 1}, 'faultcode': {'value': 226, 'length': 2, 'type': 'num', 'divide': 1}, 'pvipmtemperature': {'value': 230, 'length': 2, 'type': 'num', 'divide': 10}, 'pbusvolt': {'value': 234, 'length': 2, 'type': 'num', 'divide': 1}, 'nbusvolt': {'value': 238, 'length': 2, 'type': 'num', 'divide': 1}, 'epv1today': {'value': 266, 'length': 4, 'type': 'num', 'divide': 10}, 'epv1total': {'value': 274, 'length': 4, 'type': 'num', 'divide': 10}, 'epv2today': {'value': 282, 'length': 4, 'type': 'num', 'divide': 10}, 'epv2total': {'value': 290, 'length': 4, 'type': 'num', 'divide': 10}, 'epvtotal': {'value': 298, 'length': 4, 'type': 'num', 'divide': 10}, 'rac': {'value': 306, 'length': 4, 'type': 'num', 'divide': 1}, 'eractoday': {'value': 314, 'length': 4, 'type': 'num', 'divide': 1}, 'eractotal': {'value': 322, 'length': 4, 'type': 'num', 'divide': 1}}

[birgerj]

I made a mistake while downloading the file from github. Downloaded the html file instead of the raw content. Downloaded the correct file and now it works! I see the data being decoded and arriving in MQTT. There it is picked up by HomeAsistant GROTT integration.

Thank you very much for your time and effort!

[johanmeijer]

haha. Your welcome. If it al works I will integrate the layout in a future release (so you do not have to add a separate layout file)