Plant Daily light integral unit (mol/d·m²) changed to strange unit (mol/s·m²s)

NorbertHD commented 6 months ago

Since upgrading to homeassistant-plant Version 2024.2.0 the plant Daily light integral unit has changed to a strange unit (mol/s·m²s). The correct unit is (mol/d·m²). Also there are homeassistant.components.sensor.recorder errors in the log:

2024-03-30 15:40:10.455 WARNING (Recorder) [homeassistant.components.sensor.recorder] The unit of sensor.aloe_vera_total_ppfd_mol_integral (mol/s⋅m²s) cannot be converted to the unit of previously compiled statistics (mol/d⋅m²). Generation of long term statistics will be suppressed unless the unit changes back to mol/d⋅m² or a compatible unit. Go to https://my.home-assistant.io/redirect/developer_statistics to fix this
2024-03-30 15:40:10.455 WARNING (Recorder) [homeassistant.components.sensor.recorder] The unit of sensor.aloe_vera_dli (mol/s⋅m²s) cannot be converted to the unit of previously compiled statistics (mol/d⋅m²). Generation of long term statistics will be suppressed unless the unit changes back to mol/d⋅m² or a compatible unit. Go to https://my.home-assistant.io/redirect/developer_statistics to fix this

dli dli2

Olen commented 6 months ago

Not sure what has happened there. I sill have the correct unit here.

But it might be related to this, somehow: https://github.com/Olen/homeassistant-plant/blame/79ab5bee58b5f99b4cdef1ca6a503354b97dc9bd/custom_components/plant/sensor.py#L538

ChristophCaina commented 6 months ago

Hey, I am pretty sure, that I haven't changed anything related to the daily light integral - but in fact, I am also having the same unit of measurement in my sensors...

I will cross check my changes for the sensors I've submitted...

Olen commented 6 months ago

Strange. Mine are fine:

What are the units for the diagnostics entries?

ChristophCaina commented 6 months ago

Hi :) Even on the integration page in HACS, the screenshot is showing another value (sm², not sm²s) instead of d*m²

And the changes I've made was only related to changing TIME_SECONDS to UnitOfTime.SECONDS

which should not change anything to the unti of measurement itself in my opinion.

Also, another strange behave is, that my history stats are showing the value mol/s*m²s at least since november 2023... ?! So this would be the case way before I made the changes... if I am not wrong (I submitted the PRs in December)

EDIT: Diagnostic entities are also the same

Olen commented 6 months ago

Don't worry about the screenshot in the README. That is very old, and I did have "s" in the first iterations, I believe. But it has been "d" for ages now.

ChristophCaina commented 6 months ago

hm... as far as I can see, it is still in the sources:

UNIT_PPFD = "mol/s⋅m²"
UNIT_MICRO_PPFD = "μmol/s⋅m²"
UNIT_DLI = "mol/d⋅m²"
UNIT_MICRO_DLI = "μmol/d⋅m²"
UNIT_CONDUCTIVITY = "μS/cm"
``

and except to the UnitOfTime.SECONDS nothing has changed

class PlantTotalLightIntegral(IntegrationSensor): """Entity class to calculate PPFD from LX"""

def __init__(
    self,
    hass: HomeAssistant,
    config: ConfigEntry,
    illuminance_ppfd_sensor: Entity,
    plantdevice: Entity,
) -> None:
    """Initialize the sensor"""
    super().__init__(
        integration_method=METHOD_TRAPEZOIDAL,
        name=f"{config.data[FLOW_PLANT_INFO][ATTR_NAME]} Total {READING_PPFD} Integral",
        round_digits=2,
        source_entity=illuminance_ppfd_sensor.entity_id,
        unique_id=f"{config.entry_id}-ppfd-integral",
        unit_prefix=None,
        unit_time=UnitOfTime.SECONDS,
    )
    self._unit_of_measurement = UNIT_DLI
    self._attr_icon = ICON_DLI
    self.entity_id = async_generate_entity_id(
        f"{DOMAIN_SENSOR}.{{}}", self.name, current_ids={}
    )
    self._plant = plantdevice



Since before my change the "TIME_SECONDS" was used, this should not have changed anything...
I am confused now... what version are you running?

Olen commented 6 months ago

hm... as far as I can see, it is still in the sources:


UNIT_PPFD = "mol/s⋅m²"
UNIT_MICRO_PPFD = "μmol/s⋅m²"
UNIT_DLI = "mol/d⋅m²"
UNIT_MICRO_DLI = "μmol/d⋅m²"
UNIT_CONDUCTIVITY = "μS/cm"
``

PPFD is measured in mol/s*m², so that is correct.

Total Light Integral and Daily Light Integral is then calculated from PPFD, and those are measured in mol/d⋅m²

Basically Total Light Integral is the sum of seconds * PPFD And Daily Light Integral is a statistics sensor based on Total Light Integral that is reset every day.


class PlantTotalLightIntegral(IntegrationSensor): """Entity class to calculate PPFD from LX"""
 (...)
    self._unit_of_measurement = UNIT_DLI

And neither of these explain why you have an extra "s" int the unit mol/s⋅m²s

NorbertHD commented 6 months ago

PPFD is measured in mol/s*m², so that is correct. ... Basically Total Light Integral is the sum of seconds * PPFD ... And neither of these explain why you have an extra "s" int the unit mol/s⋅m²s

I think the confusion comes from the missing brackets.

How to interpret mol/s⋅m²s ? Is it?: mol/(s⋅m²⋅s) or (mol/s)⋅m²⋅s

I assume the system means: (mol/(s⋅m²))⋅s

Or in another notation: mol⋅s⁻¹⋅m⁻²⋅s

The s⁻¹ and the s cancel each other out and this remains: mol⋅m⁻² or mol/m²

And that's the correct unit for the x axis for the integral sum of PPFD.

antaflos commented 6 months ago

Not sure what I can contribute here, but I am seeing the same warnings about the DLI unit change:

Logger: homeassistant.components.sensor.recorder
Source: components/sensor/recorder.py:210
integration: Sensor (documentation, issues)
First occurred: 22:55:10 (11 occurrences)
Last logged: 23:00:10

The unit of sensor.office_kolbenfaden_dli is changing, got multiple {'mol/d⋅m²', 'mol/s⋅m²s'}, generation of long term statistics will be suppressed unless the unit is stable and matches the unit of already compiled statistics (mol/d⋅m²). Go to https://my.home-assistant.io/redirect/developer_statistics to fix this
The unit of sensor.office_einblatt_gross_total_ppfd_mol_integral is changing, got multiple {'mol/d⋅m²', 'mol/s⋅m²s'}, generation of long term statistics will be suppressed unless the unit is stable and matches the unit of already compiled statistics (mol/d⋅m²). Go to https://my.home-assistant.io/redirect/developer_statistics to fix this
The unit of sensor.office_einblatt_gross_dli is changing, got multiple {'mol/d⋅m²', 'mol/s⋅m²s'}, generation of long term statistics will be suppressed unless the unit is stable and matches the unit of already compiled statistics (mol/d⋅m²). Go to https://my.home-assistant.io/redirect/developer_statistics to fix this
The unit of sensor.office_bergpalme_total_ppfd_mol_integral is changing, got multiple {'mol/d⋅m²', 'mol/s⋅m²s'}, generation of long term statistics will be suppressed unless the unit is stable and matches the unit of already compiled statistics (mol/d⋅m²). Go to https://my.home-assistant.io/redirect/developer_statistics to fix this
The unit of sensor.office_bergpalme_dli is changing, got multiple {'mol/d⋅m²', 'mol/s⋅m²s'}, generation of long term statistics will be suppressed unless the unit is stable and matches the unit of already compiled statistics (mol/d⋅m²). Go to https://my.home-assistant.io/redirect/developer_statistics to fix this

ChristophCaina commented 6 months ago

based on the statistics, it seems that the change of the unit of measurement happened somehow to all sensors, including total ppfd_mol_integral statistics as well... ?!

grafik

except for MIN and MAX DLI input grafik

Still trying to see, where the new unit_of_measurement could come from - but even if I search through all files, I can only find the correct unit_of_measurement for DLI ...

ChristophCaina commented 6 months ago

another feeling I have so far is, that the change somehow came with 2024.4.0 of HomeAssistant... I'm pretty sure, that I have checked the statistics for the DLI in the last weeks more regularely, and they were working fine...

It is just for a couple of days now, that I can't see the statistic graph on the "more info" page - and need to go to the long term stats to see those.

I will setup another HA instance, running 2024.2 and 2024.3 to test against these versions over the weekend.

Olen commented 6 months ago

I have not upgraded to 2024.4 yet, but it's on my weekend plans. At least that will verify if it is a change in HA or in the Integration.

ChristophCaina commented 6 months ago

At least that will verify if it is a change in HA or in the Integration.

Even tho I would not understand what change it should be in HA... From the source, the unit_dli has been defined in the code - and will not be imported from any HA constant...

Olen commented 6 months ago

I have a hunch that HA tries to be "smart", and that since unit_time is seconds for total, it changes the unit_of_measurement from 'd' to 's' by itself

ChristophCaina commented 6 months ago

I have a hunch that HA tries to be "smart", and that since unit_time is seconds for total, it changes the unit_of_measurement from 'd' to 's' by itself

hm... I can't imagine, tbh... This would mean, HA would "manipulate" the existing constant "mol/d*m2" and replace the d with s by itself... I don't think that this is how it is working. especially, you don't use anything that would create the unit of measurement "dynamically"

Also, the Unit was always "seconds" but used the depricated TIME_SECONDS ... this was changed to the UnitOfTime.SECONDS

_DEPRECATED_TIME_SECONDS: Final = DeprecatedConstantEnum(
    UnitOfTime.SECONDS,
    "2025.1",
)
"""Deprecated: please use UnitOfTime.SECONDS."""

ChristophCaina commented 6 months ago

OK - I don't get it... The DLI sensor is still "unknown" on my test setup, but in the development tools the unit of measurement is shown in mol/d*m²

I've tested HA 2024.2.0, HA 2024.3.0 and HA 2024.4.0 Plant integration is Verison 2024.2.0

grafik

I will roll back to a previous version again and see, if it will change if the sensor gets updated

I found another strange behave while testing...

this is the area where I can change the statistics in the development tool on my productive system: grafik

It is showing, that ppfd_mol is expected to be mol/sm², total_ppfd_mol is expected to be mol/dm², just as we expect it for dli. For both entities, I have the information, that the unit_of_measurement changed to mol/s*m²s

Now, compared to my test installation, after going back to HA2024.2.0 grafik

As shown, the total_ppfd_mol is already showing mol/s*m²s - so here, we're expecting three different unit_of_measurements...

-- [Edit] even now, the sensor remains with the unit mol/d*m² with all HA versions

2024.2.0 grafik

2024.3.0 grafik

2024.4.0 grafik

So it does not seem to be the integration itself, nor a dependency directly with HomeAssistant... I am questioning myself now, if there might be another potential dependency that could cause this behave...

Olen commented 6 months ago

Just upgraded to 2024.4 and I am now 98.7% certain that the root cause of this is related to a change in HA.

First - HA tries to be smarter and better in handling measurements. These improvements has gone on for a while, and are in general a good thing. But here it fails slightly.

So what happens is:

Lets start with the diagnostics-sensors:

The PPFD-sensor (sensor.xxxxx_ppfd_mol under diagnostics in the plant device) is a regular sensors that returns a calculation of PPFD from LX. It reads the illuminance measurement from the source sensor and calculates PPFD. It is measured in mol/s⋅m². This sensor is purely a step on the way between the "lx" that is reported by most flower-sensors to get to DLI. This value increases and decreases in paralell with the "lx":

The Total PPFD-sensor (sensor.xxxxx_total_ppfd_mol_integral under Diagnostics in the plant device) is a integration sensor thst uses HAs internal procedures to calculate the integration of the PPFD-sensor above. This is an ever groing value because it is the total PPFD that the plant has ever received.

The integraiton sensors do not accept a unit_of_measurement, but will create one based on the source sensors (in this case, the PPFD-sensor) and the unit_time:

https://github.com/home-assistant/core/blob/f2fe2c45109b01d779f479a9e866f3c11ec868f7/homeassistant/components/integration/sensor.py#L352-L358

Which ends up as:

            integral_unit = f"{source_unit}{unit_time}"

So that gives the Total PPFD-sensor the unit_of_measurement "mol/s⋅m²s" as can be seen in the Diagnostics screenshot above.

I have not looked at all the details, but it seems like this code was at least modified 2 weeks ago. I don't know exactly what changed :

Ok, so we now have the reason for the "strange" unit for the Total PPFD sensor.

But how does this affect the DLI sensor, which is a UtilityMeterSensor that takes the Total PPFD and just creates daily values from it? It turns out that the UtilityMeterSensor will automatically use the source sensors unit_of_measurement instead of what is specified in the setup of the sensor:

https://github.com/home-assistant/core/blob/f2fe2c45109b01d779f479a9e866f3c11ec868f7/homeassistant/components/utility_meter/sensor.py#L527

new_state_attributes is the attributes of the source sensor when an update is received. But be aware that this only happens when the source sensors changes, so this means that if the source is not changing, you will keep the restored unit_of_measurement for a while.

I have not looked into the details here either, but it looks like this code was changed a week ago:

Again, exactly what changed have not been investigated by me.

So my conclusion is that - Yes. HA tries to be smart, and first adds an extra "s" to the Total PPFD.

And this is then picked up by the DLI sensor.

The only way to get past this, I believe, is to make HA accept that the unit_of_measurement can be set by an integration for integral-sensors and/or utilitymeter-sensors.

ChristophCaina commented 6 months ago

I am trying to get daily light Integral mol/d*m² or at least similar as a Default unit into ha - since the fyta Sensor ist already using a similar value it would make Sense in my opinion to unify the Units through all Sensors... / Integrations 🤔

Olen commented 5 months ago

This is due to https://github.com/home-assistant/core/issues/115208

Olen / homeassistant-plant

Plant Daily light integral unit (mol/d·m²) changed to strange unit (mol/s·m²s) #148