Add caching layer for inverters that go offline after dark

[robbinjanssen]

We've learned that some inverters go offline after dark. Meaning the status.js/json file is no longer reachable. Instead of throwing errors and exceptions we would like to introduce a caching layer. As discussed in the python client for this package we need to add the caching layer to this integration.

@sander76 reached out and pointed us in the right direction. Unfortunately I don't have much time right now, it would be awesome if anyone would like to give it a go!

Instructions here: https://github.com/klaasnicolaas/python-omnikinverter/discussions/24#discussioncomment-1323129

This would solve #39

[klaasnicolaas]

Let's sign this up for hacktoberfest 😄

[ronaldt80]

@klaasnicolaas is there any way to correct faulty values in the long-term statistics database? I have 2 inverters, 1 went briefly offline and back online twice today. Result is that HA Energy showed 13,000kwh production today. The previous integration had caching on so never experienced this issue before. Will probably roll back to v1 until caching is possible in V2 as well but in the meantime would be good to correct the incorrect data.

[klaasnicolaas]

Yes there is a service that I think you can use to remove incorrect data.

Regarding your issue, it is interesting to see what your inverter does after going offline and coming back online. An interruption is not bad, but it is when a 0 is passed as a value from the inverter. Can you indicate which data source you are using? Contact me via a DM in Discord (Mister Nicolaz#7558), so we can check some things.

[ELANDJEA]

@klaasnicolaas is there any way to correct faulty values in the long-term statistics database? I have 2 inverters, 1 went briefly offline and back online twice today. Result is that HA Energy showed 13,000kwh production today. The previous integration had caching on so never experienced this issue before. Will probably roll back to v1 until caching is possible in V2 as well but in the meantime would be good to correct the incorrect data.

I have seen the same happening within my environment. to me it feels like the energy_total sensor does have an issue with the "last_reset" property. As if that property is interpreted as reset today instead of long-long-timeago. In my situation it showed 13000 KwH on day one and day two it started with 26000 KwH (+ a little) within the daily energy meter.....I did not wait for the 3rd day...... I switched over to the "energy_today" property for the energy dashboard, which now acts normally for about 3 days already. (due to the daily start with 0 I asume). @klaasnicolaas I suggest to also look into the "last_reset" property that might result in the high daily readings in the dashboard. (I am not a devoper....but like the new version of the omnik plugin..working nicer than the previous version)

[klaasnicolaas]

As stated before, this has to do with your inverter sending incorrect values at certain times (we have seen this on JSON source). To determine this for sure, we need to apply some extra logging so that we can figure this out. The last_reset has nothing to do with it, it only assumes that there has been a reset due to a decreating in the total_increasing.

What source do you use and could you access it privately so we can debug this further?

[ELANDJEA]

I used the JS source for getting the data. (http:///js/status.js) if needed I can get output from that source.....guess one time output is not enough....you need at least a certain sequence?

[robbinjanssen]

Could you please open a separate issue for this, as this is something different than the caching issue :-)

[wes1993]

We have some news about this implementation?

[robbinjanssen]

Unfortunately not, currently i dont have much spare time. But i'm happy to accept a PR! :)

[freddieleeman]

I hope someone is still willing to look at this. I'm currently solving this by adding an SQL query that retrieves the latest non-"unavailable" value from the states. But it could be more convenient to have this feature in the integration.

[robbinjanssen]

I've actually started working on this :-) but ran into some issues, more updates soon!

[robbinjanssen]

Im having trouble implementing this as I cannot reproduce this with my own inverter.

Can someone tell me if their inverter becomes completely unavailable via the web as well? Or is it only the inverter and is the device still online?

[freddieleeman]

day:

night:

js: https://pastebin.com/pvr3VPVA

[timn]

The Bosswerk MI-600 goes completely dark, it never draws from the grid but only from solar power. Therefore it's offline when no sun is shining.

Would this caching layer also solve the problem that for a little while after the inverter booted up the total is showing zero before then shooting to the actual value? That's the problem that brought me here (backstory: my inverter has no internet access, because of that it does not reset its daily yield, using an utility meter with daily rotation on total yield should replace that, but it would probably produce invalid sums when there are 0 values and then totals again, I'll know for sure tomorrow, I just set this up).

[timn]

I can already confirm a similar problem: using the solar_energy_total in the energy dashboard leads to a high colum at the very beginning with the accumulated total on every day (I thought this was due to the not resetting daily yield, but it's actually the total yield, so already the energy dashboard is broken by this, and not just the utility meters I wanted to set up). This is caused by the value being zero for the first few measurements every day and then having the actual value. I can filter this with a template sensor now, but it'd be great if the integration would already do that by itself (it should be safe to always reject 0 for the total yield, that will only exist for a very short time and it should be ok not to report that).

[github-actions[bot]]

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!

[timn]

My current workaround is a custom filter such as this:

- sensor:         
    - name: Filtered Balkon Solar Energy Total
      device_class: energy
      unit_of_measurement: kWh
      state_class: total_increasing
      icon: mdi:chart-line   
      state: >                
       {% if states('sensor.balkon_solar_energy_total')|float(0) > 0 %}
         {{ states('sensor.balkon_solar_energy_total') }}
       {% else %}            
         {{ states('sensor.filtered_balkon_solar_energy_total') }}
       {% endif %}                             

This seems to work fine so far.

[freddieleeman]

I fixed it with an SQL integration:

      SELECT
        states.state
      FROM
        states
        LEFT JOIN state_attributes ON (
          states.attributes_id = state_attributes.attributes_id
        )
      WHERE
        metadata_id = (
          SELECT
            metadata_id
          FROM
            states_meta
          where
            entity_id = 'sensor.powerone_solar_energy_total'
        )
      AND states.state > 1 AND states.state != 'unknown' and states.state != 'unavailable' 
      ORDER BY
        state_id DESC
      LIMIT
        1;

But this is far from convenient. It would be great if the integration would ignore unavailable/unknown/0.

[PyroPath]

I've a rather old SolarBright 3.0kW for 12 panels on my main roof which also goes completely dark and APSystems DS3 for small cluster of XL panels on a little building in my garden. I have a kWh meter between my inverters and the grid to have a cumulative value for all solar (shame on you APSystems for reporting DC side yield stats!). This kWh meter also confirms there is 0 power draw after the sun goes down and the unit offline.

I have a tick box for using cached values, but this clears when leaving the configuration screen and thus doesn't work. It would be nice if we could get a solution for all the errors in the logbook.