Handle Buffered Data

[mriksman]

My grott was down for a few days. Now that it's back, the data from the last 3 days is being sent inbetween fresh data. However, MQTT/homeassistant-grott doesn't seem to care that it is buffered/different timestamp, so I am seeing this.

These are the buffered messages coming in. See "time": "2023-05-28T13:43:31", "buffered": "yes"

     - MQTT jsonmsg: 
             {"device": "HMG9Cxxxx", "time": "2023-05-28T13:43:31", "buffered": "yes",
             "values": {"datalogserial": "XGD6CAM1JY", "pvserial": "HMG9CAS01Z",
             "pvstatus": 1, "pvpowerin": 32580, "pv1voltage": 3230, "pv1current": 48,
             "pv1watt": 15539, "pv2voltage": 3266, "pv2current": 52, "pv2watt": 17041,
             "pvpowerout": 31930, "pvfrequentie": 4996, "pvgridvoltage": 2481,
             "pvgridcurrent": 134, "pvgridpower": 35147, "pvgridvoltage2": 0,
             "pvgridcurrent2": 0, "pvgridpower2": 0, "pvgridvoltage3": 0,
             "pvgridcurrent3": 0, "pvgridpower3": 0, "totworktime": 1002532,
             "pvenergytoday": 159, "pvenergytotal": 2522, "epvtotal": 2562, "epv1today":
             88, "epv1total": 1333, "epv2today": 73, "epv2total": 1229, "pvtemperature":
             490, "pvipmtemperature": 439}}

And the real-time data

     - MQTT jsonmsg: 
             {"device": "HMG9xxxx", "time": "2023-05-31T14:23:32", "buffered": "no",
             "values": {"datalogserial": "XGDxxxx", "pvserial": "HMG9CAS01Z",
             "pvstatus": 1, "pvpowerin": 2771, "pv1voltage": 3142, "pv1current": 4,
             "pv1watt": 1447, "pv2voltage": 3160, "pv2current": 3, "pv2watt": 1324,
             "pvpowerout": 2718, "pvfrequentie": 5004, "pvgridvoltage": 2401,
             "pvgridcurrent": 20, "pvgridpower": 6435, "pvgridvoltage2": 0,
             "pvgridcurrent2": 0, "pvgridpower2": 0, "pvgridvoltage3": 0,
             "pvgridcurrent3": 0, "pvgridpower3": 0, "totworktime": 1237066,
             "pvenergytoday": 69, "pvenergytotal": 2870, "epvtotal": 2916, "epv1today":
             36, "epv1total": 1508, "epv2today": 34, "epv2total": 1408, "pvtemperature":
             298, "pvipmtemperature": 335}}

Is there a way to

1. Ignore buffered data - don't log to MQTT?
2. Accept buffered data, but log to Home Assistant with the correct time stamp? Thanks

[mriksman]

I see the grott owner has a grott_ha extension https://github.com/johanmeijer/grott/tree/master/examples/Home%20Assistent

It drops the buffered packets.

    # Need to decode the json string
    jsonmsg = json.loads(jsonmsg)

    if jsonmsg.get("buffered") == "yes":
        # Skip buffered message, HA don't support them
        if conf.verbose:
            print("\t - Grott HA - skipped buffered")
        return 5

What benefits are there to using homeassistant-grott vs using the grott_ha.py directly from the link above?

[muppet3000]

@mriksman - Thanks for the heads up on the grott_ha extension way of handling this - I'm going to implement this in my version.

Sorry for the delayed response, I've been away on holiday and I'm now catching up on all the various bugs and tickets that have been raised.

I also suffered from this buffering issue over the last couple of days (my graphs look hilarious) so I'll be implementing the same logic myself.

As for this integration vs the one from the Grott repo - it's totally up to you. Johan has openly admitted that he doesn't really support the HA integration in his repo. Plus it's quite a primitive read-only implementation, the plans for my version of the integration is to grow it to also allow configuration of the inverters so that people can use it to configure AC charging windows etc. for battery systems.

I'll keep you posted on when I've implemented the buffered data handling (note - Home Assistant doesn't support historic insertion of data, so I'll just have to drop it).

[muppet3000]

This functionality is now available on the main branch for testing (you can select this in the re-download menu in HACS)

[muppet3000]

You should see something like this now when it encounters buffered data:

(I had a LOT of buffered data today due to a network issue at my house)

[muppet3000]

@mriksman - Has this been working correctly for you? I'm closing the ticket so that it can be included in the next release.