Missing sleep state, long wake times and errors in log with MCU2 updated cars

[markusdd]

Is there an existing issue for this?

• [X] I have searched the existing issues

What happened?

The car only switches between online and offline state, sleep does never occur. Also, the log is littered with weird API errors and the wakeup times of the car are far too frequent and long, leading to bad phantom drain. Also raised this in the discussion area: https://github.com/adriankumpf/teslamate/discussions/3083

Expected Behavior

the car goes to sleep (not offline), there are no API query errors in the log and the car actually sleeps when it is not being used or charged

Steps To Reproduce

if only I knew, but I can give my config:

• Streaming API is enabled
• apart fromt his weird behavior everything else works, drives etc are all recorded fine
• car works
• this is an 85D Model S with MCU2 upgrade, so no MCU1 energy save settings required/possible
• no other services are registered with my account except Teslamate

In the current example the below log output belongs to the last offline/online phase, error prints start happeneing at around the red arrow marker:

Relevant log output

see screen below

Screenshots

ok....and now after a long wake period we are suddenly offline....that is weird.

Additional data

No response

Type of installation

Manual

Version

v1.27.2

[JakobLichterfeld]

@micves and @JakobLichterfeld - Do we know what is causing the testing to fail? (I know nothing about Elixir, so I don't understand it)

We do have 321 automated test, to ensure TeslaMate is working as expected on every change. If the expected output changes, you need to adopt the test. If not and the test fails, it means your change breaks the system. We automatically test all this on every push, on every commit, on every pull request. And we are building separate docker images hosted on ghcr to be able to test every single pull request. This is what you guys are using above. You can see the failed test for example here: https://github.com/teslamate-org/teslamate/actions/runs/7774426112/job/21199230683

[micves]

Yes, I know why the tests fail... Now there are extra conditions before an online can be detected. This have not yet been adopted into the tests. At the moment I'm testing some changes where it does not go to internal state :start, which you saw in HA. I hope it will be simpler and make more sense. It's not commited yet, and the tests still needs to be fixed as well.

Yes, I have merged in master with release 1.28.3 I think if you subscribe to the PR here https://github.com/teslamate-org/teslamate/pull/3262 you will be notified when there are changes.

[Dhtjf]

Pulled your latest from 3262 yesterday. Other than seemingly always having an issue of sleeping the first bit after an update, no false 'starts' shown in Home Assistant and teslamate logs show picking it up. Continue to put out nice updates! Thanks!

edit: appears it is struggling to sleep. Woke up and has been suspended for about 2 hours. I do have the Home Assistant integration, with a month long poll time setup, though I disabled polling an hour ago with no change.

2024-02-18 10:20:01 2024-02-18 15:20:01.136 car_id=1 [info] Stream connecting ...
2024-02-18 10:20:02 2024-02-18 15:20:02.384 car_id=1 [info] Fake online: power is nil
2024-02-18 10:20:08 2024-02-18 15:20:08.916 car_id=1 [info] Real online detected: power is a number
2024-02-18 10:20:09 2024-02-18 15:20:09.179 car_id=1 [info] Start / :online
2024-02-18 10:21:10 2024-02-18 15:21:10.494 car_id=1 [info] Suspending logging```

[micves]

Good thing the start state doesnt show up anymore 👍

I also get these small wakeups once in a while where it starts with Fake online but get a Real online right after. This is data directly from stream API and vehicle data will only be polled after a Real online. So it shouldn't be Teslamate waking the car.

Besides that: after Tesla update to 2024.2.6 I've been having long awaken hours (5-6h) in the morning during a weekend without any driving. There is not really anything to do about it. It is suspended so vehicle_data is not polled during all this time. Hope it gets better (or maybe its just time to change the 12v battery :| )

[Dhtjf]

I messed around and went back a version of your PR which didn't result in an improvement in my sleep behavior (you're probably right that I need a new 12v). Now trying to pull and run the latest of your PR results in the follow error. The latest from the repo does successfully run, so something specific with the PR. Any ideas? /opt/app/erts-14.2.2/bin/erlexec: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /opt/app/erts-14.2.2/bin/erlexec)

[micves]

Yeah sorry, I thought I had added some build info when teslamate starts up.

I have reverted the change and a new build should be ready now. I think I have to play with that in a different PR :)

[Dhtjf]

Same result for me. Updated compose back to ghcr.io/teslamate-org/teslamate:pr-3262 and did a specific docker pull ghcr.io/teslamate-org/teslamate:pr-3262 to be sure. So maybe it wasn't your last commit but something else going on.

[JakobLichterfeld]

I think I have to play with that in a different PR :)

Sir, we don't play 😉 We test locally. Only GitHub Action related code which needs special permissions needs to be tested on a branch.

[micves]

Yes of course :) I also tested locally first where it worked fine.

But I might have jumped to conclusions that it was because of my change. I reverted and its the same problem now.

My suspicion is now on the github action that builds the image. I haven't look that much into it, but there is a new version, which might have changed something:

The right one works. The left one fails with: /opt/app/erts-14.2.2/bin/erlexec: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /opt/app/erts-14.2.2/bin/erlexec

I'm away for the weekend so I won't be looking into it, but there could be potential for a new issue :)

[JakobLichterfeld]

I'm away for the weekend so I won't be looking into it, but there could be potential for a new issue :)

Our Ghcr build is only for testing PRs.

[micves]

That is also what we are trying to do. You get that error after you pull the new PR image and try to start teslamate.

It is a new problem, it didnt do that last week and it just seems to me there is something not matching in the ghcr build image all of a sudden.

[JakobLichterfeld]

Just saying, a test build can not cause any TeslaMate issues :-)

[micves]

I think you are right that it is not a Teslamate issue. The error indicates Erlang/elixir/libc problem. It is built with GLIBC_2.34 on the GitHub action image, but the docker PR image created does not contain this version. Or am I misunderstanding something here?

We can test on an existing PR that works that was built several days ago and retrigger the build. If it still works I'll shut up about it :D

[Dhtjf]

Maybe was 2024.2.6 keeping the car awake (on 2024.2.7 now) or coincidence, but been sleeping great after using the image from commit 42ee29e that I still had, pulled a week ago.

[JakobLichterfeld]

I think you are right that it is not a Teslamate issue. The error indicates Erlang/elixir/libc problem. It is built with GLIBC_2.34 on the GitHub action image, but the docker PR image created does not contain this version. Or am I misunderstanding something here?

We can test on an existing PR that works that was built several days ago and retrigger the build. If it still works I'll shut up about it :D

Normal docker image build runs fine, see : https://github.com/teslamate-org/teslamate/actions/runs/8035109824

I can take a look into the ghcr builds next week, but it builds fine for other PRs: https://github.com/teslamate-org/teslamate/actions/runs/8034393943

[DrMichael]

Edge appears to also have the GLIBC_2.34 error.

[JakobLichterfeld]

Edge appears to also have the GLIBC_2.34 error.

will be resolved with next automatic build, as #3691 is already fixed

[USAFPride]

@micves, can you update your fork to the current?

I'm missing supercharges for some reason. The car is "offline" during those periods, but I need to test it some more.

[micves]

I'm working on it :) Discovered a bug when starting Teslamate while car is online. Not something I have seen before, so might be some unlucky timing during startup. Just wanted to test the fix a little before pushing.

I think Tesla have had some server issues the last few days. I missed a drive yesterday and can see the stream api kept reconnecting. Normal polling also timed out or reported car unresponsive.

[USAFPride]

@micves, reporting is back to normal (at least for the MCU2 UG). I am seeing long periods of 'offline' again and drive/charges/online is working correctly.

now if we can get the pesky Elixir CI / Test (Elixir 1.16.1 / OTP 26) to pass and get this integrated in the master ... :)

[oivindoh]

Is there any useful testing I can perform here to move this along? I can test with both an MCU2/HW2->MCU2/HW3 car and an MCU1/HW2->MCU2/HW3 car.

I don't run the streaming API on any of them because that messes with car position updates in the tesla app (jumping all over the globe).

[micves]

Testing the image on cars with streaming api off could be helpful to verify that nothing in that department has been broken.

Fixing the test code could also be helpful ;) Every time I think I have some time to look at it, it just ends up with a merge from main and testing it live on the car.

Car position jumping around like that sound weird and can the streaming api really cause that? Do others see similar things?

[JakobLichterfeld]

I don't run the streaming API on any of them because that messes with car position updates in the tesla app (jumping all over the globe).

Car position jumping around like that sound weird and can the streaming api really cause that? Do others see similar things?

That has been resolved by Tesla FW 2024.8.x, see #3351

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[micves]

Here's some activity

[wodnik7]

Hello

we have a Tesla Model S100D, 2017, upgraded to MCU2 since 2022.

the car never goes to sleep, as long as we have it. Other cars on the same account (MY and MX 2016 McU1) do go sleeping properly.

can it be an issue with Teslamate? Now we have 1.28.5, but it was the same on al versions.

the issue is with the car in the attachment in the bottom. The upoer one‘s issue was the caised by eevee app (now is the working ok).

no other apps do connect to the car now

[USAFPride]

@micves, can you merge the current teslamate to your fork?

[micves]

@USAFPride, just did a sync fork on github, so the images should be ready soon. I haven't updated and tested my own system yet.

@wodnik7, I don't think its a Teslamate issue, but maybe start a new issues with logs. There are some other issues and discussions you can look through. It could be Sentry mode is always on. It could be the 12V battery needing replacement.

[USAFPride]

@micves, all is going well on the updated image.

[zexpe]

I've been enjoying this fork for a few months now, not updated it to the latest version yet, but just noticed that for the first time my car is now reporting itself as being asleep rather than offline. I guess Tesla have changed something in their API?

[Jochemp88]

I have got this situation since this morning update to 2024.14.8, so it could be that Tesla has changed something in the software rather than the API, but I'm not an expert on this.

[JakobLichterfeld]

I have got this situation since this morning update to 2024.14.8, so it could be that Tesla has changed something in the software rather than the API, but I'm not an expert on this.

Since 2024.8.3 Tesla do report only offline for Intel MCUs since Spring Update for all ECUs

[zexpe]

Since 2024.8.3 Tesla do report only offline for Intel MCUs since Spring Update for all ECUs

I'm MCU2 (hence using this fork), but still on 2024.8.9 since 19th April. I started using TeslaMate on 19th February when I was on 2023.44.30.9. Ever since I've used TeslaMate my car has only reported "offline" never "asleep". It started reporting "asleep" instead of offline last night - and it's still reporting "asleep" - without any software changes to the car.

[USAFPride]

2024.14.8 I'm seeing asleep, offline, online

[zexpe]

That's interesting. It also switched from offline to asleep at the same time as mine, but mine hasn't switched backed to offline since.

[USAFPride]

@micves, can you merge the current teslamate to your fork?

Have you had any luck sorting out the errors?

[micves]

@USAFPride, I just clicked the 'sync fork' on github.com (I haven't tested on own system)

I have given the errors/pending-adjustments-in-tests a thought once in a while, but nothing more. Been too busy and as long as the PR-image (and my own dev setup) works, I tend to not prioritize it :(

I know it would make sense to get it into master at some point. Maybe I'll find some time after the summer holidays ;)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[DumSkidderik]

@USAFPride, I just clicked the 'sync fork' on github.com (I haven't tested on own system)

I have given the errors/pending-adjustments-in-tests a thought once in a while, but nothing more. Been too busy and as long as the PR-image (and my own dev setup) works, I tend to not prioritize it :(

I know it would make sense to get it into master at some point. Maybe I'll find some time after the summer holidays ;)

Now we just need @leandreeberhard to update his docker image of @micves 's code (pretty please!)

[JakobLichterfeld]

Now we just need @leandreeberhard to update his docker image of @micves 's code (pretty please!)

Why are you not using the official docker images from the PR? To do so follow: https://docs.teslamate.org/docs/development#testing-with-our-ci-which-builds-the-docker-images-automatically-per-pr

with pr-3262

[USAFPride]

Now we just need @leandreeberhard to update his docker image of @micves 's code (pretty please!)

Why are you not using the official docker images from the PR? To do so follow: https://docs.teslamate.org/docs/development#testing-with-our-ci-which-builds-the-docker-images-automatically-per-pr

with pr-3262

I just tried and there is some permission error:

image: ghcr.io/teslamate-org/teslamate/teslamate:pr-3262

✘ teslamate Error Head "https://ghcr.io/v2/teslamate-org/teslamate/teslamate/manifests/pr-3262": denied 0.7s Error response from daemon: Head "https://ghcr.io/v2/teslamate-org/teslamate/teslamate/manifests/pr-3262": denied

[JakobLichterfeld]

I just tried and there is some permission error:

Ok, strange, perhaps the build got deleted after retention days. So we either need to extend it, or just include the changes in main version, which I prefer.

[USAFPride]

@micves, is your fork currently synched?

[micves]

@USAFPride, yes a few days ago. If you subscribe to the PR here https://github.com/teslamate-org/teslamate/pull/3262 you should get e-mails whenever there are some changes.