Owl Electricity and Solar sensor measurements not able to be used with new Energy component

[shortbloke]

Home Assistant 2021.8.0 Added a new Energy Management feature.

When enabled, the list of available sensors to use for the Electricity Grid and Solar Panels doesn't include the owl sensors. The pages helpful link to: https://www.home-assistant.io/more-info/statistics/ which in turn suggests raising an issue with the integration owner to have them review: https://developers.home-assistant.io/docs/core/entity/sensor/#long-term-statistics

[petitpiton]

@shortbloke The dev (@glpatcern) needs to add some attributes to the published sensors for electricity daily consumption.

Needs two static attributes - device_class: 'energy' and state_class: 'measurement' Then also needs to check value of self._state when updating the sensor value. If the reading has gone back to zero, then set new attribute last_reset to current timestamp from the xml object. Unfortunately my Python skills are non-existent, so I can't develop a pull request!

In the meantime you can add the static attributes in home assistant via entity customisation and just a zero placeholder for the last_reset attribute. That will let you select the sensor in the Energy Management feature of HA, although at the end of the first day you'll get a negative value equal to the entire day's consumption.

[glpatcern]

Hi there,

Thanks @shortbloke and @petitpiton for the references, does not seem like a big development and I meant to upgrade my HA to see how this new Energy Management feature works. Did not have time so far... let's see the coming weekend.

[petitpiton]

That would be very helpful!

On 12 Aug 2021, at 15:05, Giuseppe Lo Presti @.***> wrote:

Hi there,

Thanks @shortbloke https://github.com/shortbloke and @petitpiton https://github.com/petitpiton for the references, does not seem like a big development and I meant to upgrade my HA to see how this new Energy Management feature works. Did not have time so far... let's see the coming weekend.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/custom-components/sensor.owlintuition/issues/16#issuecomment-897668692, or unsubscribe https://github.com/notifications/unsubscribe-auth/APFVDMKV7DZITAQAOIQQPY3T4PIMDANCNFSM5BT446PQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email.

[glpatcern]

So, I have presumably implemented what was needed and the sensor at least works as before, however I did not have time yet to look into the Energy Management dashboard. Would you be able to try and tell me? And maybe post some details of what you tried?

[petitpiton]

@glpatcern I should be able to try this tomorrow. Thanks

[shortbloke]

I've updated to the latest version and I don't see the owl sensors in the drop down when trying to configure the energy component. I've restarted HA, and the entire host. Is there something extra I need to do to get these to work? Delete/re-install the integration?

[petitpiton]

@glpatcern I think there’s a problem with the new version. I can get it to show in the drop down menu, but it’s not recording any readings. I switched from my own locally hacked version at around 6am this morning and haven’t had a reading yet…

[petitpiton]

I also would suggest that, now datetime is included in the program, the attribute for last_update could be returned as a datetime rather than an integer as it will display as a human-readable date in the attributes then.

[shortbloke]

@petitpiton Did you do anything specific to get the owl sensors to appear in the drop down?
My config is pretty simple and yet in the drop down I only get my TP Link HS110 sockets which measure per device power usage. The elements I have within HA for already monitoring power usage are continuing to work, so the core data collection remains working for me.

# Owl Energy Monitor
- platform: owlintuition
  host: 192.168.200.45
  port: 32000
  mode: triphase
  cost_icon: 'mdi:currency-gbp'
  cost_unit_of_measurement: GBP
  monitored_conditions:
    - electricity
    - solar

[glpatcern]

Thanks for trying out, I actually have the same question to @petitpiton - possibly because I did not upgrade my HA for some time, I still have to sort out lots of deprecation warnings in the logs and I don't see this Energy dashboard at all. Any clue where to look at? I guess/hope that the logs will tell more then.

Otherwise, concerning:

now datetime is included in the program, the attribute for last_update could be returned as a datetime rather than an integer as it will display as a human-readable date in the attributes then.

I'm not sure - datetime was always available, that's not an issue, yet to my understanding the last_update "system" attribute has to be a timestamp as it is the case for other integrations, or has that changed?

[petitpiton]

@glpatcern In Core update 2021.8 you should see a new energy icon (lightning bolt) in the sidebar, just below the main dashboard icon. The first time you open this, you should see a settings page to allow you to specify the sensors to record for grid/solar/individual devices. After first setup, this settings page can be found in Configuration > Energy (Grouped with 'Tags'). I attach screenshots to illustrate.

In terms of adding the Owl sensor, I must admit that I added it using my own local changes to sensor.py. I updated this to the version you released yesterday and then, although the sensor still showed up, it stopped taking readings. I deleted the sensor from the grid energy settings and then re-added it. I suspect it may have shown in the list of selectable sensors as it had already been registered as eligible by HA...

In response to the lack of any data for ~12 hours after upgrading the custom component, I reverted to my locally modified version with the addition of the official component's handling of the last_reset attribute, deleted the sensor in the energy settings again and re-added again. The readings started to come in again.

However, after leaving it running overnight (as can be seen in the screenshot) the reset doesn't seem to work as intended. I get a normal consumption figure for the first hour of the day, followed by a negative equal to the sum of the last 24 hours in the second hour. I wonder if this is an issue of naive time vs UTC time? I am in the UK and we are currently on summer time (GMT +1:00). It would appear that the sensor reset is actually happening at 00:00 GMT for me and this is local time 01:00?... I always assumed OWL would reset at 00:00 local time, but I guess this really depends on how 2SaveEnergy setup their servers. Alternatively, perhaps more likely on reflection, it may be that HA assumes it will be fed with UTC datetimes... Of course, neither may not be the cause of the issue, but it 'fits' the observations.

I notice that HA Core 2021.8.7 has become available overnight, so I will update to this latest version and see if anything changes.

I will try to find time to set up a new VM instance of HA to test the official component without having first used any other (local) version to see if it shows as selectable in the energy settings.

I don't want to post my local code (it's not my place and the coding is 'raw'), but happy to share it with you privately.

[glpatcern]

Thanks @petitpiton, that explains: I don't have the lightning icon, very likely you had it because of your local patches. Can you share them with me privately then, so I can compare?

Concerning the reset time, I also assumed it's midnight local time (at least looking at the history plots) and the integration reports it as "midnight local time". But maybe that's related to not publishing the right info in the first place, so let's check that out later.

[shortbloke]

@glpatcern If you're not seeing the Energy option, then you need to edit your configuration and ensure you have: default_config: I found that mentioned in one of the comments on the release blog for the feature.

[petitpiton]

Quick update... I have provisioned a fresh VM running HA for Linux and set it up with just the OWL integration (had to install HACS & VSCode as well). Out of the box the fresh install has the Energy Monitoring Icon in the sidebar. The OWL is configured and the sensors are receiving data. However, the Energy dashboard says I "don't have any statistics", so it's not recognising the OWL integration sensor as eligible for long term statistics... I attach the YAML from the VM config file:

# Configure a default setup of Home Assistant (frontend, api, etc)
default_config:

# Text to speech
tts:
  - platform: google_translate

group: !include groups.yaml
automation: !include automations.yaml
script: !include scripts.yaml
scene: !include scenes.yaml

# Example configuration.yaml entry
sensor:
  - platform: owlintuition
    host: 192.168.130.215         # IP address that the NetworkOWL sends to (your Home Assistant IP)
    port: 4321                # Port number you specified to the OWL data push settings
    monitored_conditions:
      - electricity

[petitpiton]

Attached screenshot of the Overview on the VM instance showing sensor data is being picked up for the OWL integration.

I will now send @glpatcern my own changes to his code that I was using before.

EDIT: Sent code to contact email

[glpatcern]

I finally got around to test this and thanks to @petitpiton 's patch it's now all working for me. Let's see if the update at midnight local time also works ;-) but I don't see why it shouldn't, now that the attributes are properly set.

As for the last_update in datetime format, I'll try and test it later.

[petitpiton]

@glpatcern Mine still isn't handling the reset properly. I wonder if you should make the total consumption the long term statistic with no reset and let HA handle the date change? It seems to handle the total consumption from my appliances ok and 2021.9 will apparently be bringing monthly and yearly monitoring... Screenshot attached to show my dashboard this morning.

EDIT: My Bad, there is only a daily consumption sensor in the OWL data!

[shortbloke]

It seems to be working for me. Not sure if your environment/timezone is different.

Version | core-2021.8.7 -- | -- Installation Type | Home Assistant OS Development | false Supervisor | true Docker | true user | root Virtual Environment | false Python Version | 3.9.6 Operating System Family | Linux Operating System Version | 5.10.17-v8 CPU Architecture | aarch64 Timezone | Europe/London

Sensors used:

• Grid Consumption: sensor.owl_intuition_electricity_today
• Return to grid: sensor.owl_intuition_solar_exported_today
• Solar Production: sensor.owl_intuition_solar_generated_today

[petitpiton]

@shortbloke Great to see it's working for you. So I appear to have a local issue! My system seems to be the same as yours...

## System Health

version | core-2021.8.7
-- | --
installation_type | Home Assistant OS
dev | false
hassio | true
docker | true
user | root
virtualenv | false
python_version | 3.9.6
os_name | Linux
os_version | 5.10.17-v8
arch | aarch64
timezone | Europe/London

Can you check your time format setting in your user profile to see if it matches mine?

[shortbloke]

My profile settings are different (don't recall changing them, probably upgraded from my original HA install many years ago)

[shortbloke]

Hmm, so I didn't have owl data reporting yesterday which is why I think it worked for me. I just added some TP-Link devices and you can see that some are reporting negative values, as it appears it's not looking at the usage "today" but for the last 24hrs. Might be related to: https://github.com/home-assistant/core/issues/54633

[glpatcern]

It actually worked for me as well, with no issue at midnight local time (CEST). At the same time I have a simpler setup with no solar panels, but for what you suggest the situation could well evolve with the upcoming HA version. Let's see...

[shortbloke]

Well it's still sort of working but with the latest 2021.9.1 version of HA, I'm getting an error indicating a new state class should be being used: expected state class 'total_increasing' Developer blog: https://developers.home-assistant.io/blog/2021/08/16/state_class_total

[glpatcern]

Thanks Martin, indeed this is due to https://github.com/home-assistant/core/pull/54523. Will patch the code accordingly.

[glpatcern]

I left the patched code for the night but things got worse, as the backend now seems to "sum the sum" up. It may well be I did not understand the referenced PR but I don't see what's wrong in my patch. @shortbloke could you try it for your setup with the solar panels to have another case?

[petitpiton]

The patched code is working fine for me…

[shortbloke]

@glpatcern I'm testing now, but before the latest update is was going wrong, with values just increasing. Too early to say if the latest change fixes it, but looks more promising. Need to see what happens overnight and for generation during the day.

[shortbloke]

All looking good with the latest version @glpatcern

[glpatcern]

Glad to hear that! I still have to fix my setup, obviously it's something on my side, not in the code ;-)

[shortbloke]

Well it looks like I have some other issue, not sure if it's the integration or something else. As I've not had any data Solar "exported" or "exporting" data for a day. Power consumption and Solar Generation sensors are working fine.

[psl3]

I have this now showing 6 entities in Lovelace and used consumption today, solar generated today and solar exported today in the Energy component. These are now showing values in the graphs but think these are incorrect. I think the owl setup needed for the owl app to work (as well as the 3 entities for current data) is from a clamp on the solar generated and a clamp on the total used in the house (either from solar or grid). However, the HA energy feature is expecting total Solar Generated and total electricity from the grid. So although I have used roughly 25kwh today and solar has generated 12kwh, in the HA Energy Distribution (3 circles graph) it is showing 25kwh from the gride, 12KWh from Solar so things I have used 37KWh total (i.e. has added in the Solar twice). I could move the clamp to the grid cable but then the other entities are wrong and the owl app is wrong. Anyone else seen this? Is there an option anywhere to say the electricity reading is actually into the fusebox rather than from the grid?

[shortbloke]

@psl3 I have the Owl setup with 2 clamp meters one measuring just the solar feed and the other going to the consumer unit, which sounds the same as you. A Type 1 Installation:

I'll admit I was pleased to see the graphs showing data, but hadn't actually considered if the data shown was correct. It seems to be this is a deficiency in the Energy component that it doesn't account for how/where the measurements are being taken.

However, I've had a crack at solving this. Using the (new) template component (rather than the old sensor -> template, which doesn't allow state_class to be set) we can create our own sensor:

template:
  - sensor:
    - name: "house_grid_energy_today"
      unit_of_measurement: "kWh"
      state_class: total_increasing
      device_class: energy
      state: >
        {% set p1 = states('sensor.owl_intuition_electricity_today_p1') | float %}
        {% set p2 = states('sensor.owl_intuition_electricity_today_p2') | float %}
        {%- if float(p1) - float(p2) > 0 -%}
        {{ float(p1) - float(p2) }}
        {%- else -%}
        0
        {%- endif -%}
      attributes:
          last_reset: >
            {{ state_attr('sensor.owl_intuition_electricity_today_p1', 'last_reset') }}

    - name: "house_grid_energy_now"
      unit_of_measurement: "W"
      state_class: measurement
      device_class: energy
      state: >
        {% set p1 = states('sensor.owl_intuition_electricity_power_p1') | float %}
        {% set p2 = states('sensor.owl_intuition_electricity_power_p2') | float %}
        {%- if float(p1) - float(p2) > 0 -%}
        {{ float(p1) - float(p2) }}
        {%- else -%}
        0
        {%- endif -%}

Where p1 is the connection to the consumer unit and p2 is the connection monitoring the solar generation. If the value has gone negative, then it should report zero, so it avoids double accounting for exporting energy. I've only just added this sensor to my setup, so need to wait for HA to do the hourly update of the Energy component.

Hopefully, HA will provide a means in the future to say that one needs to be a sum of the other, or deducted from the other.

[psl3]

Wow - looks promising - have pasted your code and changed for my entities (sensor.owl_intuition_electricity_today and sensor.owl_intuition_solar_generated_today) and added. Also made a lovelace card so tomorrow when I have solar I can see if the numbers add up. Well done and thank you for generating - hope this is a solution.

Just one thing - if your default entities have the P1 and P2 etc., I wonder if your owl component is different - I think I have the latest version provided here or is this something you have edited.

[shortbloke]

@psl3 I'm currently experimenting with the P1, P2 sensors vs the electricity and solar specific sensors, as I've spotted some inconsistencies. Just trying to work out where they are coming from. So yes, using the P1 P2 or the specific electricity and solar versions of the sensors shouldn't matter.

If I can demonstrably prove this is working then I'll raise a bug against HA to see if they are willing to make it configurable how the grid and solar interact. I expect they have designed it for a means of measuring solar (current clamp, meter, data from inverter) and a smart meter providing grid consumption data, which is a bit different to the owl, so not sure if the template sensor will be the way forward for us.

Or if the owl component, should implement a sensor that tries to provide grid data, but I'm not sure it can know exactly how the owl has been set up with the various current clamp installation options.

[psl3]

@shortbloke As feedback from your additional code - Grid data looks good obviously just a calculation but is below is much easier to absorb by seeing total used and proportion of grid and solar energy. The owl data also matches the owl app so all good. What I am confused about is when these exact same entities are added to the energy component - it seems to get them wrong.

[shortbloke]

@psl3 I suspect I need to change that sensor template, seeing the value drop to zero causes issues, which may also be related to what you're seeing.

I think if either sensor was yet to return a value, then it would have returned 0, which especially for the total_increasing sensor would be a problem. This is more common than I thought (see #18) especially after a HA restart. Though the issue #18 will also impact the accuracy of the reporting of usage throughout the day.

I've updated the sensor to return nan (not a number), instead of zero, will see if that fixes things.

template:
  - sensor:
    - name: "House Grid Energy Now"
      unit_of_measurement: "W"
      state_class: measurement
      device_class: energy
      state: >
        {% if is_state('sensor.owl_intuition_electricity_power', 'unknown') or is_state('sensor.owl_intuition_solar_generating', 'unknown') %}
          nan
        {% else %}
          {% set elec = states('sensor.owl_intuition_electricity_power') | float %}
          {% set solar = states('sensor.owl_intuition_solar_generating') | float %}
          {%- if float(elec) - float(solar) < 0 -%}
            0
          {%- elif float(elec) - float(solar) >= 0 -%}
            {{ float(elec) - float(solar) }}
          {%- endif -%}
        {%- endif -%}

    - name: "House Grid Energy Today"
      unit_of_measurement: "kWh"
      state_class: total_increasing
      device_class: energy
      state: >
        {% if is_state('sensor.owl_intuition_electricity_today', 'unknown') or is_state('sensor.owl_intuition_solar_generated_today', 'unknown') %}
          nan
        {% else %}
          {% set elec = states('sensor.owl_intuition_electricity_today') | float %}
          {% set solar = states('sensor.owl_intuition_solar_generated_today') | float %}
          {% if float(elec) - float(solar) >= 0 %}
            {{ float(elec) - float(solar) }}
          {%- endif -%}
        {% endif %}

[glpatcern]

Interesting trick, I concur the template logic could be incorporated in HA. A minor correction (maybe not so relevant but still): the device_class of a measurement (power or anything not incrementing over time) is actually power, not energy.

[shortbloke]

I was getting unexpected spikes and also the total usage decreasing with the template I suggested above. I think I've ironed things out now, but only time and sunshine will tell.

template:
  - sensor:
    - name: "Owl Grid Energy Now"
      unit_of_measurement: "W"
      state_class: measurement
      device_class: power
      state: >
        {% if is_state('sensor.owl_intuition_electricity_power', 'unknown') or is_state('sensor.owl_intuition_solar_generating', 'unknown') %}
          nan
        {% else %}
          {% set elec = states('sensor.owl_intuition_electricity_power') | float %}
          {% set solar = states('sensor.owl_intuition_solar_generating') | float %}
          {% if (float(elec) - float(solar)) < 0 %}
            0
          {% elif (float(elec) - float(solar)) > 0 %}
            {{ float(elec) - float(solar) }}
          {% endif %}
        {% endif %}

    - name: "Owl Grid Energy Today"
      unit_of_measurement: "kWh"
      state_class: total_increasing
      device_class: energy
      state: >
        {% if is_state('sensor.owl_intuition_electricity_today', 'unknown') or is_state('sensor.owl_intuition_solar_generated_today', 'unknown') %}
          nan
        {% else %}
          {% set elec = states('sensor.owl_intuition_electricity_today') | float %}
          {% set solar = states('sensor.owl_intuition_solar_generated_today') | float %}
          {% set last_grid_today = states('sensor.owl_grid_energy_today') | float %}
          {% if (float(elec) - float(solar)) > 0 %}
            {% if (float(elec) - float(solar)) > float(last_grid_today) %}
              {{ float(elec) - float(solar) }}
            {% else %}
              {{ float(last_grid_today) }}
            {% endif %}
          {% endif %}
        {% endif %}

I've also raised an issue (well an enhancement request really) against Home Assistant: https://github.com/home-assistant/core/issues/56135

[wcavanagh]

I wonder if anyone can help me at all. I am trying to get my Owl P1 and P3 to report into the Energy Dash. However it is not showing any data. If I look at the sensors in admin I can see the Kwh readings so that is correct. I have pasted my config.

#For Energy Consumption Dashboard.
template:
  - sensor:
    - name: house_energy_P1
      state: "{{ states('sensor.owl_intuition_electricity_power_p1') }}" 
      unit_of_measurement: 'kWh'
      state_class: measurement
      device_class: energy
      attributes:         
          last_reset: '1970-01-01T00:00:00+00:00'

  - sensor:
    - name: house_energy_P3
      state: "{{ states('sensor.owl_intuition_electricity_power_p3') }}" 
      unit_of_measurement: 'kWh'
      state_class: measurement
      device_class: energy
      attributes:         
          last_reset: '1970-01-01T00:00:00+00:00'

AM I missing some something?

[glpatcern]

@wcavanagh please check the template as posted earlier in this issue, it seems you have a much smaller and incomplete declaration. Also, please check the recently opened issue, in case you're running the latest HA release.

In general, I'd suggest you to post a question to the community.home-assistant.io discourse service, or to open a new GitHub issue if it seems an issue with the component, rather than posting to an already closed issue.

[nickloman]

I wonder if anyone can help me at all. I am trying to get my Owl P1 and P3 to report into the Energy Dash. However it is not showing any data. If I look at the sensors in admin I can see the Kwh readings so that is correct. I have pasted my config.

Hi @wcavanagh

I think I got this working by modifying your template a bit to use 'today' aggregated values rather than the real-time readings.

  - sensor:
    - name: house_energy_P1
      state: "{{ states('sensor.owl_intuition_electricity_today_p1') }}"
      unit_of_measurement: 'kWh'
      state_class: measurement
      device_class: energy
      state_class: total_increasing
  - sensor:
    - name: house_energy_P2
      state: "{{ states('sensor.owl_intuition_electricity_today_p2') }}"
      unit_of_measurement: 'kWh'
      state_class: measurement
      device_class: energy
      state_class: total_increasing
  - sensor:
    - name: house_energy_P3
      state: "{{ states('sensor.owl_intuition_electricity_today_p3') }}"
      unit_of_measurement: 'kWh'
      state_class: measurement
      device_class: energy
      state_class: total_increasing
  - sensor:
    - name: house_energy_total
      state: "{{ states('sensor.owl_intuition_electricity_today_all') }}"
      unit_of_measurement: 'kWh'
      state_class: measurement
      device_class: energy
      state_class: total_increasing