Huge spike in export every morning

[Hufflepuff-source]

I am now able to see the import and export but every morning, I see a huge spike in the export totalling to the day before or something.

Any idea on how to fix this? Also, is there a sensor to get the daily export instead of getting the total export increasing variable?

[sammcewan]

I get this too. Came here to ask the same question!

[sammcewan]

Had a poke around to see how this works and it seems api side. I'd say this number is meant to just be rough with little emphasis on accuracy. Probably need to explore an alternative calculation approach or find another API.

[sammcewan]

Giving this a try to try and solve the imported problem. Won't solve the exported issue which seems to go haywire when the inverter gets turned off.

    - sensor:
      - name: "Power Import Filtered Total"
        unique_id: "sensor.power_import_filtered_total"
        device_class: energy
        unit_of_measurement: 'kWh'
        state_class: total_increasing
        state: >
            {% set previousFilteredValue = states('sensor.power_import_filtered_total') %}
            {% set liveValue = states('sensor.homekit_x_import') %}

            {% if liveValue | float < previousFilteredValue | float(0) %}
                {{ previousFilteredValue }}
            {% else %}
                {{ liveValue }}
            {% endif %}

      - name: "Power Export Filtered Total"
        unique_id: "sensor.power_export_filtered_total"
        device_class: energy
        unit_of_measurement: 'kWh'
        state_class: total_increasing
        state: >
            {% set isPVRunning = states('sensor.pv_power') | float > 0 %}
            {% set previousFilteredValue = states('sensor.power_export_filtered_total') %}
            {% set liveValue = states('sensor.homekit_x_export') %}

            {% if liveValue | float < previousFilteredValue | float(0) %}
            {{ previousFilteredValue }}
            {% else %}
                {% if isPVRunning %}
                {{ liveValue }}
                {% else %}
                {{ previousFilteredValue }}
                {% endif %}
            {% endif %}

[sammcewan]

Okay think I've fixed it by just not recording if the inverter is off (updated the above). This is using the goodwe integration. Imagine theres a way to work this back into the sensor, will monitor for now.

[philipbrennan]

ok so just recording it if its an increase and if pv_power which is if there is power being currently produced?

[philipbrennan]

ah, you've got a homekit plus inverter. I've homekit only. I think given the pattern I could just base it on the time of day. or possibly discard updates which are rising beyond a certain rate.

[sammcewan]

Yeah I humoured using a delta check keen for any ideas you might have. The last thing for me to solve is why my template still allows that step down.

[philipbrennan]

what I would try is, store the values you want to compare into variables. then compare them.

[philipbrennan]

as Im thinking that its reading the sensor state, making decision but then reading it again and it may have changed .

[sammcewan]

Ace, updated my snippet.

[TimSoethout]

Hi all, good work on this. Also feel free to add this to the python code of the component (via pull request) if that is easier. That would also fix this for all users. :)

[philipbrennan]

Yup, I haven't updated my local yet at all but will issue PR once its done ( not sure if I can make it work for homekit / inverter users in all combinations though ) To be honest what's been annoying me more recently is the SEMS api being very slow, and the data itself sometimes not appearing for several hours. I'd be ok with an occasional outage for updates but this is for days at a time. I don't think its fit for purpose - pity as the hardware seems fine. Has anyone tried intercepting the comms from the homekit to the cloud? or portscanning it? For an inverter you can already get data via direct connection.

[sammcewan]

I played with adding it to the sensor but my understanding was you'd need to manually query the database to obtain the previous value. Any pointers on this as would love to crack open a pr with my solution?

For what it's worth capturing the variables at the start seems to have fixed my issue with import too so I've got decent confidence in this workaround.

Agree about it not being fit for purpose, it came as a surprise when the solar provider set us up with it rather than solar analytics. I'll have a crack at port scanning it over the weekend.

[Denifia]

I have a similar issue where my import and export values are wrong each morning but am taking a different approach to solve it.

I found that the only values that were wrong are Totals_sum and Totals_buy. After a period of time (a few mins to a few hours), the values would correct themselves. During my issue (left side) the values are much lower than expected, then a minute later (right side) has the correct values.

As the code is using Home Assistant's state class of total_increasing, I decided to use the daily values and let them reset to zero each morning.

For HomeKit's Export, I'm now using data["Charts_sell"] and for the Import I'm using data["Charts_buy"].

I can make a PR if others want the option of using daily values instead of the running total.

[philipbrennan]

I have a similar issue where my import and export values are wrong each morning but am taking a different approach to solve it.

I found that the only values that were wrong are Totals_sum and Totals_buy. After a period of time (a few mins to a few hours), the values would correct themselves. During my issue (left side) the values are much lower than expected, then a minute later (right side) has the correct values.

As the code is using Home Assistant's state class of total_increasing, I decided to use the daily values and let them reset to zero each morning.

For HomeKit's Export, I'm now using data["Charts_sell"] and for the Import I'm using data["Charts_buy"].

I can make a PR if others want the option of using daily values instead of the running total.

I think you mean you are using the daily values summed via total increasing to be the running total.. yeah makes sense. you get a smooth graph then?

[Denifia]

Yes, that's probably what I mean - I'm yet to find out as I only put this fix in today and cleared all my historical stats. Tomorrow morning I'll find out if it works but my logs show me it's got a decent chance of being the fix for me.

[Denifia]

I quickly exported the attributes with the below query to get a comparison of the Charts_buy/sell vs Totals_buy/sell.

SELECT 
  JSON_EXTRACT(state_attributes.shared_attrs, '$.Charts_buy') as Charts_buy, 
  JSON_EXTRACT(state_attributes.shared_attrs, '$.Charts_sell')  as Charts_sell,
  JSON_EXTRACT(state_attributes.shared_attrs, '$.Totals_buy')  as Totals_buy, 
  JSON_EXTRACT(state_attributes.shared_attrs, '$.Totals_sell')  as Totals_sell,
  states.last_updated
FROM states
join state_attributes on states.attributes_id = state_attributes.attributes_id 
where entity_id = "sensor.homekit_<redacted>"
order by state_id
limit 1000

[Denifia]

Using Charts_buy/sell instead of Totals_buy/sell fixed the issue for me. No spike, happy days!

If I get time I might look into pulling historical data from the sems api into HA's statistics table so I can backdate my fix.

[sammcewan]

Epic! Would be super keen for a PR, even something rough to set this up 🙏

[philipbrennan]

@sammcewan and @TimSoethout The PR here will be in the form of templates. Need a total increasing sensor that uses the daily total instead of the overall. One for import one for export. But from what I can remember the templates in this repo use the older sensor format. I basically started from scratch with mine.So they could do with a bit of a revamp anyway. I've made the change locally to charts_buy and charts_sell , hopefully smooth graphs from now on.

[Denifia]

Added what I've done locally in PR #76 so others can replicate if they wish.

[TimSoethout]

I merged and release the PR. Hope this fixes it for the HomeKit users. Thanks for the support.