"Interpolate values" looks weird

[RubenKelevra]

Version of home_assistant

2023.12.0b1

Version of the custom_component

2.0.13

Describe the bug

I've enabled the option "Interpolate values" while setting up the weather and activated a plot of the solar radiation. I've noticed that, while the steps every x hour(s) are gone, the interpolation isn't smooth, but tries only to smooth out half of the time it should. Leading to a half smooth, half stepped graph which looks kinda weird.

I think as solar radiation is kinda smooth on a non-cloudy day the interpolation issue is the most obvious on it.

Debug log

No debug log due to privacy concerns.

[FL550]

I'll have to look into this. Thanks for reporting!

[FL550]

As this only occurs in specific points in time, I think it might have something to do with an update of the forecast data.

The algorithm I use is a linear interpolation, so nothing special. I'll have to further investigate.

[RubenKelevra]

Hey @FL550,

Thanks for looking into it.

If I look at the graph it looks to me like there's a shift of half of the interval forward of the interpolated values.

So half of the interval gets no interpolation, then the second half of the interval gets the first part of the interpolation and then the new value creates a 100% gradient in the graph.

[FL550]

Hm, I can't confirm this with my setup. I only have a step one time a day, at 11 o'clock. Here is a closer look of one day but this is the same for me for other days:

Can you please have a closer look at your data so we can track this down?

[FL550]

@RubenKelevra any update on this?

[RubenKelevra]

Hey @FL550, this graph doesn't look like it's the same one I'm having troubles with.

I'm seeing the issue especially on the solar radiation one.

Can you check this graph on your end? Maybe the issue is limited to this one?

[RubenKelevra]

Well, I think it's also visible on the "cloudyness" graph. But it's so much random data, I'm not sure about it.

Solar radiation however should be smooth, but it's stepped with smoothed out parts in between:

[FL550]

@RubenKelevra very strange, I don't have this in my sun irradiance sensor, only a slight step at 11:00 which is due to a data update:

Can you please tell me your station id, so I can have a closer look?

[t785634]

My temperature (and also the other values) graph looks very strange, too. Interpolation ist activated.

[FL550]

@t785634 @RubenKelevra Sorry for the late reply. Do you have the hourly update enabled? If yes, this might be the cause for this issue.

[RubenKelevra]

@t785634 @RubenKelevra Sorry for the late reply. Do you have the hourly update enabled? If yes, this might be the cause for this issue.

I think you're referring to the interpolation of values, right?

[FL550]

@RubenKelevra Yes, the interpolation and the hourly update combined.

[RubenKelevra]

@RubenKelevra Yes, the interpolation and the hourly update combined.

Yeah sure. That's what I'm using.

[t785634]

@t785634 @RubenKelevra Sorry for the late reply. Do you have the hourly update enabled? If yes, this might be the cause for this issue.

Yes, me too. That's what I'm using. Is there a way to use both with correct interpolation?

[FL550]

Ok, then this is the error source. However, right now I have no idea how to fix this. Let me explain the problem.

DWD provides the data as hourly values. Lets call the value for the current hour x0 and the value for the next hour x1. For the interpolation, I use a simple linear function where I gradually shift between the values x0 and x1 based on the seconds between these timestamps: x0 + (x1 - x0) * seconds_difference

Every new hour, x0 is replaced by x1 and the next hourly value is loaded from the cached forecast into x1 (Normal data update occurs every 6 hours).

However, when the data is refreshed every hour, x0 is updated to a completely new value, which is most likely different from the old "predicted value in one hour". This produces this visible shift in the data.

So when for example the old predicted values for the temperatures at 12:00 is 12.0 and at 13:00 is 13.0, the computed value at 12:59 accorrding to the above formula is approx. 12.98. If at 13:00 new data is loaded from DWD because of the hourly update and the new, corrected value is now 12.9, the graph drops from 12.99 to 12.9 instead of rising to the old 13.0.

[RubenKelevra]

I'll give the source code a look later. Thanks for your explanation.

[FL550]

I don't think that this will resolve the core issue. However, you are always invited to have a look for improvements.

[RubenKelevra]

Ok, then this is the error source. However, right now I have no idea how to fix this. Let me explain the problem.

DWD provides the data as hourly values. Lets call the value for the current hour x0 and the value for the next hour x1. For the interpolation, I use a simple linear function where I gradually shift between the values x0 and x1 based on the seconds between these timestamps: x0 + (x1 - x0) * seconds_difference

Every new hour, x0 is replaced by x1 and the next hourly value is loaded from the cached forecast into x1 (Normal data update occurs every 6 hours).

However, when the data is refreshed every hour, x0 is updated to a completely new value, which is most likely different from the old "predicted value in one hour". This produces this visible shift in the data.

So when for example the old predicted values for the temperatures at 12:00 is 12.0 and at 13:00 is 13.0, the computed value at 12:59 accorrding to the above formula is approx. 12.98. If at 13:00 new data is loaded from DWD because of the hourly update and the new, corrected value is now 12.9, the graph drops from 12.99 to 12.9 instead of rising to the old 13.0.

Agreed that's not simple to fix.

The simplest solution would be to ignore the next value, if the hour has already started. And use the updated values for the next hour afterwards.

The next simplest solution would be to update the data right before the hour mark, in addition to the fix above. This ensures a low delay for rapidly updated data, as only ~60 minutes delay is introduced, not avg 90 minutes.

However this would add punctual load onto the DWD infrastructure, as all home assistant instances update at roughly the same time, instead of (presumably currently) just randomly 60 minutes after the boot update.

[FL550]

The simplest solution would be to ignore the next value, if the hour has already started. And use the updated values for the next hour afterwards.

This would mean, that even if there is more precise data available, this data is held back.

The next simplest solution would be to update the data right before the hour mark, in addition to the fix above. This ensures a low delay for rapidly updated data, as only ~60 minutes delay is introduced, not avg 90 minutes.

Check for new data is done every 10 minutes, so if the shift occurs during the hour, this means there was a new update from DWD. So nothing can be done about the 90 min delay. DWD updates the data roughly ever 60 minutes.

However this would add punctual load onto the DWD infrastructure, as all home assistant instances update at roughly the same time, instead of (presumably currently) just randomly 60 minutes after the boot update.

This is already the case, as soon as new data is available, all clients fetch these new data nearly at the same time.

[t785634]

Are there any updates?

[FL550]

Please see above, there is currently no good solution for this.

[RubenKelevra]

Please see above, there is currently no good solution for this.

Actually I think there is.

The issue is different than we thought:

Say we got the following values:

08:00 is 1 09:00 is 2

An update is run at 08:30 and the updated value for 09:00 is 2.1.

Then the graph should be increasing until 08:30 with 100% slope and then the slope should be slightly steeper as the value for 09:00 is now higher.

Instead the graph is jumping to 2.1 at the time the update is run.

This is causing this weird steps, if my analysis is correctly.

[FL550]

@RubenKelevra Yes, this could smoothen the jumps a bit at the cost of precision. This is however something I can write in the notes so every user is aware of this.

The bigger problem I see is with larger corrections of data like in the image of @t785634 . With your idea implemented this would still result in a zigzag line, as the interpolated value is already above (or below, but just stick to the rising part of temperature in the morning) the new value.

So continuing your example, this would be the case if the new updated value for 09:00 is 1.4 at 08:30.