energy consumption values decrease every day at 22:30 (CET)

[FlorianWilk]

Europe Hyundai (KONA)

v2.16.3

I have a decrease of "sensor.kona_total_energy_consumption" and "sensor.kona_total_energy_regeneration" - values every day at around 22:00-22:40 (CET). This makes EnergyMonitoring of "sensor.kona_total_energy_consumption" in my EnergyDashboard pretty hard (often values over the day then are negative)

There is no error in the logs. The values i receive after around 22:00-22:40 are just lower than the values i get over the day, which results in a negative peak in the energy dashboard. I even setup a brand new HomeAssistant-Instance and added the kia_uvo integration. Everything works fine until 22:00-22:40. Then i receive lower values than i had over the day.

Expected behaviour would be a static increase of these values.

Here you can see the lower value at exactly 22:30:

Same with regeneration:

Values over 5 days - every day has these lower values at 22:00-22:40 with different amounts:

I hope my description makes sense.

[Bubo08]

Are you seeing the same in your Kia Connect App?

[FlorianWilk]

Are you seeing the same in your Kia Connect App?

In the App i do not see any possibiliy to see those statistics in Detail. I only have a Menu for "Energy usage", which then shows me stats and a graph for a predefined timespan (in my case 10. Nov - 09.dec). It doesn't even show any values in the graph, so it's difficult to tell, if this already happens in the "Hyundai-Core-Stack".

But: Even though i can't see the values, i see that the chart / proportions of the chart are different from what i see in my EnergyManagement in HA. I cropped i "interesting" part for better visibility:

Chart in my Hyundai APP 1. Dec - 9. Dec

Same timespan in HA / EnergyDashboard.

The first 3-4 days seem to be pretty similar in their proportions but then days 5-7. dec are ~the half of day 4. Looking at the 24h Dashboard View in HA, i can see that those days (5-6) have "very high" negative peaks.

4. dec (low negative peak):

5-7. dec (higher negative peak)

This could explain the differences of their proportions in the Hyundai APP vs HA App.

[Bubo08]

In the first few pictures you are showing Wh, in the new ones you are showing kWh. Was the sensor in the first pictures wrongly configured? In home assistant you can edit history values. If you remove the negative values, does it then look correct? If so you may filter out negativ values in home assistant. This is not a solution more a work around.

[FlorianWilk]

The sensor still shows "Wh", when i open it in Settings->Entities->kona_total_energy_consumption view:

The values in the dashboard shows kWh - no manual config from my side, guess it's just smart :) - the increase over the day (until 22:xx) seem to be correct:

In the pictures above there is no negative peak yet, because it's 20:05 now. So the negative value will show up in about 2 hours.

As described this is a fresh HA install. In my previous HA Installation i tried several things to remove this negative peak using DevTools->Statistics->Adjust Sum. But then i had another problem: Yes, the negative peak at 22:xx was gone, but showed up the next day when i did the first ride with the car. It seems as if the value after the decrease was the new "truth" for "total_energy_comsumption". What i try to understand: Does this value "total_energy_consumption" come from the Hyundai API itself (which then would be a bug in this API itself, right?), or is it somehow calculated in the Backend/Lib of the Hyundai-Kia-Connect Integration. And if it was a bug in the Hyundai API, why does not everbody who uses this API have this issue? Since i do not have a "Total Energy Consumption"-Value in the Hyundai APP, i cannot say if the value given from the Integration is right or wrong, which makes it hard to identify the source of the bug. But since it is the only value from this integration which handles energy-consumption, it's the only value i can use in the EnergyDashboard. Not sure if the HyundaiAPI provides more information about energy. I even thought about writing a script or filter, which just stores it's own "total comsumption" value, ignores any decrease and just sums up the increases, but ... this feels so "hacky" and i was hoping that i could just get "the truth" from the API itself. :)

[Bubo08]

Only do make sure: Did you increase the log level to debug like descripted at the end in https://github.com/Hyundai-Kia-Connect/kia_uvo?

[FlorianWilk]

Yes i did. Interesting, i just verified it in the logs and noticed a message i didn't notice before:

Entity sensor.kona_total_energy_consumption from integration kia_uvo has state class total_increasing, but its state is not strictly increasing. Triggered by state 466542 (469781.0) with last_updated set to 2023-12-07T22:03:08.457592+00:00. Please create a bug report at https://github.com/fuatakgun/kia_uvo/issues December 8, 2023 at 10:35:10 PM

Here is the last log, no decrease yet, so this propably will be the values before the next decrease in a few minutes: The highlighted value (476811) is also the value the "kona_total_energy_consumption" shows (see screenshot below)

2023-12-09 22:30:43.276 DEBUG (SyncWorker_9) [hyundai_kia_connect_api.KiaUvoApiEU] hyundai_kia_connect_api - get_driving_info responseAlltime {'retCode': 'S', 'resCode': '0000', 'resMsg': {'drivingInfo': [ {'drivingPeriod': 0, 'totalPwrCsp': 476811, 'motorPwrCsp': 316771, 'climatePwrCsp': 81098, 'eDPwrCsp': 59890, 'batteryMgPwrCsp': 0, 'regenPwr': 204144, 'calculativeOdo': 2909}, {'drivingPeriod': 1, 'totalPwrCsp': 5544, 'motorPwrCsp': 3683, 'climatePwrCsp': 943, 'eDPwrCsp': 696, 'batteryMgPwrCsp': 0, 'regenPwr': 2373, 'calculativeOdo': 33.825581395348834}], 'drivingInfoDetail': [ {'drivingPeriod': 1, 'drivingDate': '202312', 'totalPwrCsp': 69763, 'motorPwrCsp': 45210, 'climatePwrCsp': 16052, 'eDPwrCsp': 6950, 'batteryMgPwrCsp': 0, 'regenPwr': 19774, 'calculativeOdo': 301}, {'drivingPeriod': 1, 'drivingDate': '202311', 'totalPwrCsp': 167326, 'motorPwrCsp': 102092, 'climatePwrCsp': 39566, 'eDPwrCsp': 19750, 'batteryMgPwrCsp': 0, 'regenPwr': 69387, 'calculativeOdo': 923}, {'drivingPeriod': 1, 'drivingDate': '202310', 'totalPwrCsp': 148440, 'motorPwrCsp': 106010, 'climatePwrCsp': 18913, 'eDPwrCsp': 18020, 'batteryMgPwrCsp': 0, 'regenPwr': 62136, 'calculativeOdo': 1006}, {'drivingPeriod': 1, 'drivingDate': '202309', 'totalPwrCsp': 91282, 'motorPwrCsp': 63459, 'climatePwrCsp': 6567, 'eDPwrCsp': 15170, 'batteryMgPwrCsp': 0, 'regenPwr': 52847, 'calculativeOdo': 679}]}....

[FlorianWilk]

one information i forgot: As i said the negative peak always occurs around 22:xx-23:xx. In the docu of the integration was written, that the default config does no force update between 22:00 and 6:00, which drove me to the idea, that this error occurs because of a wrong cached value. So i changed the config to "no force refresh start hour: 23" and "no force refresh finish hour: 0". But that didn't have any effect regarding the negative peak.

[Bubo08]

I'm also new to this api. I'm trying to find out why my EV9 is not delivering any information as you might see in my bug. So I am setting up the environment and try to debug the problem.

Back to your problem: My understanding is, that the drivingPeriod 0 is the total drivingPeriod': 0, 'totalPwrCsp': 476811

and then drivingPeriod 1 is the last since reset or since the last charging drivingPeriod': 1, 'totalPwrCsp': 5544

and then a list of detailed drives per month follows drivingPeriod': 1, 'drivingDate': '202312', 'totalPwrCsp': 69763

Could you please also past the log from 2023-12-09 23:00?

[FlorianWilk]

I was planning to show you the logs of yesterday evenings negative peak as soon as it occurs, but guess what: It didn't happen. yesterday was the first day without negative peak. I guess the bug knows we are talking about him. As soon as the peak shows up, i will publish logs here (before and after).

[FlorianWilk]

Here we go - since i did not drive the car today, the totalPwrCsp of 476811 from yesterday is still valid. Last log before negative peak/decrease of totalPwrCsp (and all attributes, see comment below):

2023-12-10 21:50:16.029 DEBUG (SyncWorker_9) [hyundai_kia_connect_api.KiaUvoApiEU] hyundai_kia_connect_api - get_driving_info responseAlltime {'retCode': 'S', 'resCode': '0000', 'resMsg': {'drivingInfo': [{'drivingPeriod': 0, 'totalPwrCsp': 476811, 'motorPwrCsp': 316771, 'climatePwrCsp': 81098, 'eDPwrCsp': 59890, 'batteryMgPwrCsp': 0, 'regenPwr': 204144, 'calculativeOdo': 2909}, {'drivingPeriod': 1, 'totalPwrCsp': 5544, 'motorPwrCsp': 3683, 'climatePwrCsp': 943, 'eDPwrCsp': 696, 'batteryMgPwrCsp': 0, 'regenPwr': 2373, 'calculativeOdo': 33.825581395348834}], 'drivingInfoDetail': [{'drivingPeriod': 1, 'drivingDate': '202312', 'totalPwrCsp': 69763, 'motorPwrCsp': 45210, 'climatePwrCsp': 16052, 'eDPwrCsp': 6950, 'batteryMgPwrCsp': 0, 'regenPwr': 19774, 'calculativeOdo': 301}, {'drivingPeriod': 1, 'drivingDate': '202311', 'totalPwrCsp': 167326, 'motorPwrCsp': 102092, 'climatePwrCsp': 39566, 'eDPwrCsp': 19750, 'batteryMgPwrCsp': 0, 'regenPwr': 69387, 'calculativeOdo': 923}, {'drivingPeriod': 1, 'drivingDate': '202310', 'totalPwrCsp': 148440, 'motorPwrCsp': 106010, 'climatePwrCsp': 18913, 'eDPwrCsp': 18020, 'batteryMgPwrCsp': 0, 'regenPwr': 62136, 'calculativeOdo': 1006}, {'drivingPeriod': 1, 'drivingDate': '202309', 'totalPwrCsp': 91282, 'motorPwrCsp': 63459, 'climatePwrCsp': 6567, 'eDPwrCsp': 15170, 'batteryMgPwrCsp': 0, 'regenPwr': 52847, 'calculativeOdo': 679}]},

Here the log at 22:20 including the decreased value of totalPwrCsp: (difference -3130Wh)

2023-12-10 22:20:16.071 DEBUG (SyncWorker_6) [hyundai_kia_connect_api.KiaUvoApiEU] hyundai_kia_connect_api - get_driving_info responseAlltime {'retCode': 'S', 'resCode': '0000', 'resMsg': {'drivingInfo': [{'drivingPeriod': 0, 'totalPwrCsp': 473681, 'motorPwrCsp': 315154, 'climatePwrCsp': 80601, 'eDPwrCsp': 59070, 'batteryMgPwrCsp': 0, 'regenPwr': 202372, 'calculativeOdo': 2889}, {'drivingPeriod': 1, 'totalPwrCsp': 5572, 'motorPwrCsp': 3707, 'climatePwrCsp': 948, 'eDPwrCsp': 694, 'batteryMgPwrCsp': 0, 'regenPwr': 2380, 'calculativeOdo': 33.98823529411764}], 'drivingInfoDetail': [{'drivingPeriod': 1, 'drivingDate': '202312', 'totalPwrCsp': 69763, 'motorPwrCsp': 45210, 'climatePwrCsp': 16052, 'eDPwrCsp': 6950, 'batteryMgPwrCsp': 0, 'regenPwr': 19774, 'calculativeOdo': 301}, {'drivingPeriod': 1, 'drivingDate': '202311', 'totalPwrCsp': 167326, 'motorPwrCsp': 102092, 'climatePwrCsp': 39566, 'eDPwrCsp': 19750, 'batteryMgPwrCsp': 0, 'regenPwr': 69387, 'calculativeOdo': 923}, {'drivingPeriod': 1, 'drivingDate': '202310', 'totalPwrCsp': 148440, 'motorPwrCsp': 106010, 'climatePwrCsp': 18913, 'eDPwrCsp': 18020, 'batteryMgPwrCsp': 0, 'regenPwr': 62136, 'calculativeOdo': 1006}, {'drivingPeriod': 1, 'drivingDate': '202309', 'totalPwrCsp': 88152, 'motorPwrCsp': 61842, 'climatePwrCsp': 6070, 'eDPwrCsp': 14350, 'batteryMgPwrCsp': 0, 'regenPwr': 51075, 'calculativeOdo': 659}]},

Whats also interesting: Not only totalPwrCsp but many other attributes/values have been decreased - but...not all of them. f.ex. "calculativeOdo" by -20km

I searched the Log for this message from the comment above: "Entity sensor.kona_total_energy_consumption from integration kia_uvo has state class total_increasing, but its state is not strictly increasing. " But the last time it was logged was at 7.12, so 4 days ago. I would have expected this message now, too.

i also searched the log for this new value of totalPwrCsp of "473681" - i thought maybe this set of values might be a cached result from the past, but i did not have these values before.

btw: Thanks Bubo08 for your support/interest in this issue!

[fuatakgun]

@FlorianWilk , is your timezone +2 currently?

[FlorianWilk]

@FlorianWilk , is your timezone +2 currently?

The timezone is UTC+1 currently. Do you think it's a time-issue?

[Bubo08]

For me it looks like the data source of the Connect app is the problem. Perhaps, you can check if the Vehicle report -> Driving info in the app is also changing at that time, even if it is hard to compare. The other possibility would be to try to force and update at 22:00.

[FlorianWilk]

I did a force update yesterday. Regarding the app-compare: As you can see in the post 2 days before, i tried to compare the two energy-consumption charts and it seems as if they WOULD be equal if there was no negative peak. So to me it seems as if the HyundaiAPP does not have this decrease of values. I just tried to compare other values from the HyundaiAPP with the values given from the Integration. Comparing the total values is hard, because the App only shows the last timespan. But nevertheless, i compared those then:

The result of get_driving_info response30d does match the infos given in the HyundaiAPP (f.ex. totalCwp of 180983 and regenPwr 62188). And it seems to provide correct infos / without decrease before and after the decrease in the values of get_driving_info responseAlltime:

2023-12-10 21:50:16.147 DEBUG (SyncWorker_9) [hyundai_kia_connect_api.KiaUvoApiEU] hyundai_kia_connect_api - get_driving_info response30d {'retCode': 'S', 'resCode': '0000', 'resMsg': {'drivingInfo': [{'drivingPeriod': 0, 'totalPwrCsp': 180983, 'motorPwrCsp': 112429, 'climatePwrCsp': 44412, 'eDPwrCsp': 19270, 'batteryMgPwrCsp': 0, 'regenPwr': 62188, 'calculativeOdo': 882},

2023-12-10 22:20:16.203 DEBUG (SyncWorker_6) [hyundai_kia_connect_api.KiaUvoApiEU] hyundai_kia_connect_api - get_driving_info response30d {'retCode': 'S', 'resCode': '0000', 'resMsg': {'drivingInfo': [{'drivingPeriod': 0, 'totalPwrCsp': 180983, 'motorPwrCsp': 112429, 'climatePwrCsp': 44412, 'eDPwrCsp': 19270, 'batteryMgPwrCsp': 0, 'regenPwr': 62188,

So the values in get_driving_info response30d do not change, while the values in get_driving_info responseAlltime are different from one update to another (at around 22-23:00). This is really strange, isn't it?

[Bubo08]

Do you also have the Kia Connect app version 2.1.15 and a very new car? Then it might be that you have the same issue as I have because the app is not using the same api anymore at least for new cars.

[FlorianWilk]

The App is called "Hyundai Bluelink Europe" in my case and is in version 2.0.16, the car is from 2020 (if i am not wrong). We have it for ~3.5years now. I don't use the App very often, because it doesn't provide the infos i am interested in - i really like the HA Integration. I've been using the integration for a while now and I've never noticed any errors. It was only when I recently started to display the energy values in the EnergyDashboard (including EnegryCosts) that I realised I had negative peaks, because there are days, where the negative value is larger than the consumption over the day which results in negative costs :)

[FlorianWilk]

because the app is not using the same api anymore at least for new cars.

Is there any new API for your car / new cars? Maybe their API is just buggy - this could be a reason why they don't show up the total values in their App?

[Bubo08]

You could filter out the negativ values by using a template filter in Home Assistant like here: https://community.home-assistant.io/t/ignore-sensor-values-outside-of-a-certain-acceptable-range/2471?page=2

[FlorianWilk]

Yes, i thought about that, but filtering wouldn't be enough (at least in my understanding) as the new value (after the decrease) is the new base for the upcoming updates. if i would just filter out values, which are lower than the ones before, i would lose informations about new trips until we reach the old (higher) total again. The only possibiliy i see is to have a filter/sensor, which has it's own energyTotal value, where only positive deltas of the original energyTotal updates are added. This would mean that "my total energy" would be quite different than what the Hyudai API says more and more over time. And to be honest i was hoping to find a solution to get "the truth" from the API itself... as it should know best :)

[FlorianWilk]

Yesterday i had another large negative peak of 5.63kWh which brought the totalConsumption back to the value of ~3 days ago.

I tried to find out where the 5.63kWh comes from as this won't be random, but without success. It's driving me nuts! :) It would be so cool / helpful to have a working consumption-value for the dashboard as we could find out how much energy/money we spend for driving this car.

[Bubo08]

You could try the Node Red integration https://flows.nodered.org/node/node-red-contrib-bluelinky. It would be interesting if you get the same values. If not you would at least have a work around.

[FlorianWilk]

You could try the Node Red integration https://flows.nodered.org/node/node-red-contrib-bluelinky. It would be interesting if you get the same values. If not you would at least have a work around.

Good point! I will give it a try!

[FlorianWilk]

ok, interesting:

My HA EnergyDashboard / total_EnergyConsumption of today:

The negativ peak is 3.49kWh. So the EnergyConsumption should be 2.35+3.49=5.84kWh

Here is what BlueLinky gives me as result for today:

So it seems to be correct! Thanks for the hint @Bubo08 ! Bluelinky doesn't seem to have a "total of all times energy consumption"-Value, just history of days/month, so i have to figure out, how to sum all this up into a absolute total value. I thnk UtilityMeter can do this, not sure.

edit: I was wrong, BlueLinky has a total which is 466317 in my case. This is exactly what the kona_total_energy_consumption value shows (after the negative peak). so ... well, i guess it's a problem in the HyundaiAPI

[FlorianWilk]

Ok, well, so after two days with NodeRed BlueLinky in comparison, here is what i found out

NodeRed BlueLinky also returns a decreased cummulated consumption value:

Yesterday:

Today:

This is exactly what the HA Integration shows. So i think the Problem is the HyundaiAPI here and that i will close this Issue here, right?

Interesting is, that the DAILY value of BlueLinky seems to be correct, even after the negative Peak occurs. So i guess i will give it a try using the daily total in combination with "UtilityMeter" to sum the daily usage into my own total value.

[Bubo08]

I think they have a new API which is not backward compatible. Therefore, your app is using the newer version and is showing the correct values. In my opinion the way forward is to sniff KIA Connect and find out what protocol they are using and then adapt the code. As I think my problem has the same root cause I am trying to sniff my connection but I am still strugling with the setup to do so.

[FlorianWilk]

Oh cool! Wireshark and android Emulator? Would be interesting to join you on this journey :) any chance to contact you?

[FlorianWilk]

Closing this issue because the problem seems to be the Hyundai API, not this integration.

[fuatakgun]

I have used geny motion and nox earlier as android emulator and followed this process: https://docs.telerik.com/fiddler-everywhere/capture-traffic/capture-from-android

[Bubo08]

@FlorianWilk Sure, everybody is welcome :-) I never sniffed an android app so it might be quite some work. The question is how do we share best our work.

@fuatakgun, @FlorianWilk I am still struggling with the https connection and with the certificate. I have now a connection between an old handy with KIA connect on it and HTTP Toolkit but HTTPS is not working, yet.

Failed trials on Windows 11 were:

• Android Studio with different x86_64 Pixel 7 emulator => KIA connect can not be installed
• Android Studio with x86 Pixel 7 emulator => KIA connect can be installed but is crashing
• Android Studio with arm64 => AVD is not starting
• Android 8 handy with patched apk or xapk installed => Startup failed with error message "The app was forged. ..."

Next step is to root the handy and try again.

[FlorianWilk]

If you use a real physical handy/smartphone, you'll need to have access to the certificate and i guess it's just possible on rooted devices. Thats why i used an emulator running on my machine, so that you can intercept/read all the traffic. But yes, there might be some issues when installing the app of interest in the emulator. But i guess that could be avoided by using another emulator as @fuatakgun suggested - not sure. I didn't have time yet to try it, but i guess i will find some time during the days. The post from @fuatakgun above seems to be interesting, indeed. Do you have Discord or something similar and can send me an invite to have a chat?

[cdnninja]

Bluelinky discord is where we sometimes chat on that.

Charles proxy plus the box emulator worked for ssl last time I tried.

[Bubo08]

Bluelinky discord is where we sometimes chat on that.

Charles proxy plus the box emulator worked for ssl last time I tried.

Might I ask if you have the newest KIA Connect 2.1.15 on your emulator? Because even with my rooted device I am failing. The app is crashing with "The app exited because the device was rooted".

[fuatakgun]

I would suggest;

• root the emulator
• install the root certificates
• unroot it (not sure if this is even a real word)

[Bubo08]

Thanks for the suggestion.

I was not able to install the app on the emulator and the physical device can not be unrooted as far as I know. It would be intesting if somebody has a working setup to sniff the traffic.

[Bubo08]

I created a Discord Channel https://discord.gg/gRMpqcgu. I will document my work status there. Everybody who wants to help is invited :-)

[rsegers]

I was the original author of this feature. It was working previously, but somewhere along the way this was changed in https://github.com/Hyundai-Kia-Connect/hyundai_kia_connect_api and not in the Bluelink API.

The current value of 'Total Energy Consumption' is the cumulative of the past 92 days (as noted here: https://github.com/Hyundai-Kia-Connect/kia_uvo/pull/308#issuecomment-1152989365) In the original PR this value was 'Monthly Energy Consumption' and would contain the value of responseAlltime["resMsg"]["drivingInfoDetail"][0] which is the current month. But now it contains responseAlltime["resMsg"]["drivingInfo"][0], which is the 92 days cumulative and thus decreases. At least, if on day 93 a trip was made that consumed more energy than the energy consumption of today.

My proposal is to either

1. revert back to the current month consumption (which is included in the Bluelink API); or
2. construct a sensor that indeed contains the total energy consumption

I am willing to work on option 1, but I'm open for option 2.

[lordtamo]

Hi I vote for the option 1. I’m interested to know how it cost to drive the car. If by containing the total consumption you mean « consumed minus regerated », I think the sensor being less pertinent.

by the way : thanks a lot for the work. I’m using it every day, more than the Hyundai Bluelink app.