Missing Sensors - Githubissues

I have a Growatt SPF 5000 ES and have homeassistant-grott setup and seemingly working however only have 4 sensors showing and updating correctly as can be seen in this screenshot:

I am using https://github.com/egguy/ha-addons-edge by @egguy for the Grott integration and it is receiving data from my data logger no problem and sending it through to MQTT as can be seen in the following Grott log:

\x47\x72\x6f\x77\x61\x74\x74\x47\x72\x6f\x77\x61\x74\x74\x47\x72\x6f\x77\x61
         \x74\x74\x47\x72\x6f\x77\x61\x74\x74\x47\x72\x6f\x77\x61\x74\x90\x3e
     - Grott automatic protocol detection
     - Grott data record length 359
     - layout   :  T060104SPF
     - no matching record layout found, try generic
     - Record layout used :  T06NNNNSPF
     - Growatt data decrypted V2
     - Grott Growatt data decrypted
     - Growatt plain data:
         02340006015f01044444443042474e30324600000000000000000000000000000000000000005
         a584b30434b383051330000000000000000000000000000000000000000170a17151514030000
         002c0002000000000000000000000000000000000000044c00000f82000000000000000013140
         0640f370000000008fc1388000000e100bc00160000000000000000009d00a60011001f000000
         000000000000000000000000004e89002d005909ce0e830000000000ef0000071300000000000
         00000000000000000007f000000300000022d000000000000005700000000046000000f820000
         05e6000005e6000005e600000000000000160000001f000000b9000001710000005a008600000
         00110e600000000123b0e6c06fe08a713880a4dfd6a089704990004013800fa01160000000000
         00000000000000000000000000000000000000000000000000000000000000000000000000000
         000000000000000000000e479
     - Growatt new layout processing
         - decrypt       :  True
         - offset        :  6
         - record layout :  T06NNNNSPF

     - Grott data record date/time processing started
     - date-time:  2023-10-23T21:21:20
     - Grott values retrieved:
         -  datalogserial        :  DDD0BGN02F
         -  pvserial             :  ZXK0CK80Q3
         -  pvstatus             :  2
         -  vpv1                 :  0.0
         -  vpv2                 :  0.0
         -  ppv1                 :  0.0
         -  ppv2                 :  0.0
         -  buck1curr            :  0.0
         -  buck2curr            :  0.0
         -  op_watt              :  110.0
         -  pvpowerout           :  110.0
         -  op_va                :  101632.0
         -  acchr_watt           :  0.0
         -  acchr_VA             :  0.0
         -  bat_Volt             :  48.8
         -  batterySoc           :  100
         -  bus_volt             :  389.5
         -  grid_volt            :  0.0
         -  line_freq            :  0.0
         -  outputvolt           :  230.0
         -  pvgridvoltage        :  230.0
         -  outputfreq           :  50.0
         -  invtemp              :  22.5
         -  dcdctemp             :  18.8
         -  loadpercent          :  2.2
         -  buck1_ntc            :  15.7
         -  buck2_ntc            :  16.6
         -  OP_Curr              :  1.7
         -  Inv_Curr             :  3.1
         -  AC_InWatt            :  0.0
         -  AC_InVA              :  0.0
         -  faultBit             :  0
         -  warningBit           :  0
         -  faultValue           :  0
         -  warningValue         :  0
         -  constantPowerOK      :  2510
         -  epvtoday             :  23.9
         -  pvenergytoday        :  23.9
         -  epvtotal             :  181.1
         -  eacCharToday         :  0.0
         -  eacCharTotal         :  12.7
         -  ebatDischarToday     :  4.8
         -  ebatDischarTotal     :  55.7
         -  eacDischarToday      :  0.0
         -  eacDischarTotal      :  8.7
         -  ACCharCurr           :  0.0
         -  ACDischarWatt        :  112.0
         -  ACDischarVA          :  397.0
         -  BatDischarWatt       :  151.0
         -  BatDischarVA         :  151.0
         -  BatWatt              :  151.0
     - MQTT jsonmsg: 
             {"device": "ZXK0CK80Q3", "time": "2023-10-23T21:21:20", "buffered": "no",
             "values": {"datalogserial": "DDD0BGN02F", "pvserial": "ZXK0CK80Q3",
             "pvstatus": 2, "vpv1": 0, "vpv2": 0, "ppv1": 0, "ppv2": 0, "buck1curr": 0,
             "buck2curr": 0, "op_watt": 1100, "pvpowerout": 1100, "op_va": 1016320,
             "acchr_watt": 0, "acchr_VA": 0, "bat_Volt": 4884, "batterySoc": 100,
             "bus_volt": 3895, "grid_volt": 0, "line_freq": 0, "outputvolt": 2300,
             "pvgridvoltage": 2300, "outputfreq": 5000, "invtemp": 225, "dcdctemp": 188,
             "loadpercent": 22, "buck1_ntc": 157, "buck2_ntc": 166, "OP_Curr": 17,
             "Inv_Curr": 31, "AC_InWatt": 0, "AC_InVA": 0, "faultBit": 0, "warningBit":
             0, "faultValue": 0, "warningValue": 0, "constantPowerOK": 2510, "epvtoday":
             239, "pvenergytoday": 239, "epvtotal": 1811, "eacCharToday": 0,
             "eacCharTotal": 127, "ebatDischarToday": 48, "ebatDischarTotal": 557,
             "eacDischarToday": 0, "eacDischarTotal": 87, "ACCharCurr": 0,
             "ACDischarWatt": 1120, "ACDischarVA": 3970, "BatDischarWatt": 1510,
             "BatDischarVA": 1510, "BatWatt": 1510}}
     - Grott MQTT topic used : energy/growatt
     - MQTT message message sent
     - Grott Send data to PVOutput disabled 
     - Grott Send data to Influx disabled 
     - Grott extension processing disabled

Grott configuration is set as follows:

Is there something I am missing here?

Please can you try using the 'main' version of this integration, I have been implementing these sensors over the last week or so and haven't been able to do a full release with them yet. Please let me know how you get on with them and if they are accurately named etc.

If you're not sure how to use the 'main' version of the integration, let me know and I'll give you the steps. (You can do it all through HACS)

Ah ok, I did install main however I then ran the update Home Assistant was requesting I run to v1.1 which obviously then rolled it back to a previous version. I have not installed main again with any further updated and 39 sensors are now showing out of 51 in the Grott logs. So much better improvement. Is there a quick was I can see which ones are not showing?

Ah ok, I did install main however I then ran the update Home Assistant was requesting I run to v1.1 which obviously then rolled it back to a previous version. I have not installed main again with any further updated and 39 sensors are now showing out of 51 in the Grott logs. So much better improvement. Is there a quick was I can see which ones are not showing?

Main will show everything I have a mapping for. What you'd need to do is to compare the grott logs that you posted before against all of the ones shown on the UI. If there's anything you're missing that you want mapped through you just need to tell me what it's grott name is and what it's human readable name is and I'll add it in.

I have an SPH3600 and use latest version of grott, alongside mqtt and HACS, all in seperate docker containers. I have the 'main' branch of this repo because it looks like some sensors were updated after the 1.1.0 release (although I could be wrong on that but it's not wholly important to my point). Everything works fine and I've loads of sensors in HA. All good.

However, I cannot find exact (or close to accurate) matches for the 6 key metrics that Growatt use on their web portal (highlighted in green in screenshot below) or in the ShinePhone App.

growatt-portal-key-metrics

Likely I'm being thick, but outside of Solar none exactly match between the Growatt portal and grott/HA sensors created from your integration.

I believe they map as listed below, with the exception of Import from Grid

System Production = sensor.oocxxxxxxx_system_output_self_consumption_export_today_calculated
Solar Generation = sensor.oocxxxxxxx_pv_all_energy_today_calculated
Load Consumption = sensor.oocxxxxxxx_load_consumption_energy_today
Self Consumption = sensor.oocxxxxxxx_load_consumption_energy_pv_battery_today_calculated
Export to Grid = sensor.oocxxxxxxx_export_to_grid_energy_today
Import from Grid = sensor.oocxxxxxxx_import_from_grid_energy_today

Import from Grid in the Growatt portal doesn't include any AC Battery Charge energy. If you put the inverter into Battery First mode overnight for some off-peak cheap energy into the batteries, the portal will not add that energy to the Import from Grid value. I asked Growatt and they said they do not treat this as import, had their reasons, and it won't be changing. Say you have imported from the grid 2kWh, but have run Battery First and imported 5kWh, instead of 7kW the Growatt portal shows just the 2kWh. In that scenario, the sensor sensor.oocxxxxxxx_import_from_grid_energy_today would hold 7kWh but I can't find a suitable sensor relating to import or calculated that would hold the 2kWh.

I have three questions

Are all 6 of the key metrics from the Growatt portal already part of your custom/calculated sensors?
Could you please update a readme somewhere to define each of the calculated sensors you create and the calculation behind each of them?
Would you consider creating an additional calculated sensor to equate to what the Growatt portal shows as Import from Grid value?

In an ideal scenario, there would be a way to track the Today and Total for Import from Grid values across Actual and Battery AC Charged within Home Assistant.

I've had a stab at this myself with a template, but I'm not convinced I've got this right.

  - name: Import from grid (Actual) (Today)
    state: "{{ ( states('sensor.oocxxxxxxx_import_from_grid_energy_today')|float(2) - states('sensor.oocxxxxxxx_battery_ac_charge_energy_today')|float(2) )|round(1) }}"
    state_class: "measurement"
    unit_of_measurement: "kWh"
    icon: "mdi:home-import-outline"
    unique_id: "custom_import_from_grid_actual_today"

The reason I am asking all this is that if you are viewing the Energy Trend area on the Growatt portal, those 6 sensors are used for active graphs, but also for Day, Month, and Year summary views. I've been using the monthly summary view since my solar system was installed back in May 2022, and would really like to reproduce those 6 key metrics but in Home Assistant, with a view to dumping Growatt completely and just using grottserver plus your integration.

This is a sample of what I get as a message to MQTT from grott

{
    "device": "OOCxxxxxxx",
    "time": "2023-10-28T15:44:48",
    "buffered": "no",
    "values": {
        "datalogserial": "JPCxxxxxxx",
        "pvserial": "OOCxxxxxxx",
        "pvstatus": 5,
        "pvpowerin": 1100,
        "pv1voltage": 1844,
        "pv1current": 2,
        "pv1watt": 502,
        "pv2voltage": 1842,
        "pv2current": 3,
        "pv2watt": 676,
        "pvpowerout": 4846,
        "pvfrequentie": 5003,
        "pvgridvoltage": 2450,
        "pvgridcurrent": 22,
        "pvgridpower": 4838,
        "pvgridvoltage2": 0,
        "pvgridcurrent2": 0,
        "pvgridpower2": 0,
        "pvgridvoltage3": 0,
        "pvgridcurrent3": 0,
        "pvgridpower3": 0,
        "totworktime": 92117398,
        "eactoday": 64,
        "pvenergytoday": 64,
        "eactotal": 63483,
        "epvtotal": 66602,
        "epv1today": 18,
        "epv1total": 28841,
        "epv2today": 30,
        "epv2total": 37586,
        "pvtemperature": 286,
        "pvipmtemperature": 250,
        "pvboosttemp": 264,
        "bat_dsp": 527,
        "eacharge_today": 61,
        "eacharge_total": 6473,
        "batterytype": 1,
        "uwsysworkmode": 5,
        "systemfaultword0": 0,
        "systemfaultword1": 0,
        "systemfaultword2": 0,
        "systemfaultword3": 0,
        "systemfaultword4": 0,
        "systemfaultword5": 0,
        "systemfaultword6": 0,
        "systemfaultword7": 0,
        "pdischarge1": 4200,
        "p1charge1": 0,
        "vbat": 529,
        "SOC": 91,
        "pactouserr": 0,
        "pactousertot": 0,
        "pactogridr": 6,
        "pactogridtot": 6,
        "plocaloadr": 5300,
        "plocaloadtot": 5300,
        "spdspstatus": 5,
        "spbusvolt": 2978,
        "etouser_tod": 81,
        "etouser_tot": 18978,
        "etogrid_tod": 2,
        "etogrid_tot": 30649,
        "edischarge1_tod": 51,
        "edischarge1_tot": 23185,
        "eharge1_tod": 79,
        "eharge1_tot": 20008,
        "elocalload_tod": 99,
        "elocalload_tot": 58108
    }
}

Hi @dg362 thank you for the detailed information, it's rare that you get someone provide all the possible detail needed to investigate an issue. Sorry for the delayed response, I've been away on holiday and I'm slowly catching up.

I'll attempt to respond to each of your points now (please let me know if there's anything I've missed):

Anything that has 'calculated' in it's variable name is one of my calculated sensors
I can update the readme to describe each of the calculated sensors, however in the meantime I'm hoping you can parse this and get the answers you're after: https://github.com/muppet3000/homeassistant-grott/blob/main/custom_components/grott/sensors/sensors_calculated.py
I think the sensor you're looking for is "Load Consumption Energy (AC) - Today (Calculated)" from the calculated sensors: https://github.com/muppet3000/homeassistant-grott/blob/main/custom_components/grott/sensors/sensors_calculated.py#L43C14-L52 which is calculated by Subtracting the 'Battery AC Charge' value from the 'Import from Grid' value. (It looks like there is an equivalent of these two values for 'total' so I could implement the 'total' equivalent of this sensor if you'd like).

Let me know what you think and I'll assist further if needed.

I trust you had a proper break on your holiday and are well-rested in both body and mind. And you had some fun along the way, as all good holidays should be a mixture of relaxing and fun.

Thanks for providing the details, which against points 2 and 3 I think close these out. It would be beneficial to have both the Today and Total to remove the need to create something else to track the total (I only used the virtual meter because it's built-in and provides a simple way to just keep increasing the value by each day amount with a periodic reset).

For point 1, I did look through the various calculations and source .py files to see if I could work out the mapping. Because a lot of Growatt owners will be used to the 6 key metrics from the Growatt monitoring site, accepting you have to recognise some are related to graphs and some not, might I humbly suggest that these be added into a readme somewhere so that anyone wanting to see 'equivalent' data has an easy way to do this without some elements of guesswork.

I use the term 'equivalent' because it's interesting (well sort of) that the data for these sensors within HA is never quite the same as how Growatt calculates what they show on the monitoring portal. Besides what they state for the hover-over calculations, it does seem there are either some minor adjustments made to the underlying data or more likely some rounding going on at different levels to what HA is doing.

Before I had any awareness of Grott and your integration, the Growatt portal was all I had to work from. So I based a lot of my record keeping of monthly statistics on those 6 values because they give you a sense of the overall platform and usage.

All that said, this is all 'nice to have' and most certainly not an issue or something that needs to be fixed.

Thanks for the feedback. I'm happy to provide whatever additions that people see useful. In order to do the best possible job of implementing it, can I ask for the following from you?

A bullet point list of the 'total' statistics you'd like so that I can see if it's possible to create them (bonus points if you can tell me the name of the 'today' equivalent if I currently provide that
A bullet point list of the specific values on the Growatt UI/Portal that you'd like me to provide documentation/name mappings for so I can make sure I've covered them accurately.

For the first bullet, to have Totals to match Today, this would ideally (if possible) be against

sensor.oocxxxxxxx_load_consumption_energy_ac_today_calculated
sensor.oocxxxxxxx_load_consumption_energy_pv_battery_today_calculated
sensor.oocxxxxxxx_load_consumption_energy_pv_today_calculated
sensor.oocxxxxxxx_self_consumption_solar_battery_energy_today_pvenergytoday
sensor.oocxxxxxxx_system_output_self_consumption_export_today_calculated

For the second bullet this is driven by the main interface front page of the Growatt/UI and is the green highlighted items in the picture below. Noting that Solar Generation is also requested but already covered.

growatt-portal-key-metrics

But making it clear that this relates to the Graph calculations and not the Self-consumption calculation methods (i.e. the green boxed area in the screenshot below).

growatt-portal-key-metrics-chart

For completeness, that would translate to requesting a mapping that shows which sensor matches to

System Production: (Ppv + Pdischarge1 - Pcharge1) / 1000
Photovoltaic Output: Ppv / 1000
Load Consumption: (PPv + Pdischarge1 + PacToUser - PacToGrid - Pcharge1) / 1000
Self-Consumption: (PacToGrid > 0 (Load Consumption) else (System Production))
Exported to Grid: PacToGrid / 1000
Imported from Grid: PacToUser / 1000

@muppet3000 I have noticed the lack of having bunch of Today related sensors like Load Consumption, and Self-Consumption in the registered sensors, I did a quick search in the issues and came across this topic. I believe the sensors @dg362 have mentioned are very useful. May we get you to incorporate them into the integration?

muppet3000 / homeassistant-grott

Missing Sensors #60