bimmerconnected / bimmer_connected

🚘 Library to query the status of your BMW or Mini from the ConnectedDrive portal
Apache License 2.0
362 stars 79 forks source link

Error fetching bmw_connected_drive data: Client error '403 Quota Exceeded' #485

Closed jprouty closed 1 year ago

jprouty commented 1 year ago

Describe the issue

I am new to Home Assistant and have been running the BMW connected integration for the last 10 days without issue (other than this likely unrelated issue). Today I started seeing the error below spuriously in my home-assistant.log. I have not inteacted with any of the BMW HA entities directly in the past 24 hrs, or opened the BMW App on my phone.

The frequency of the error seems to be: 10m, 15m, 20m. Based on the common denominator, I assume the BMW integration has a polling interval of 5m. Is there a setting to reduce the polling interval?

In the meantime, I disabled 'Polling' in the HA BMW Connectect integration under "system settings". This should allow my account to cool down and refill quota / karma.

The severity of this issue is medium, as it disables all HA actions within the HA frontend (due to entities being unavailable) and prevents the offical MyBMW app from properly refreshing the vehicle status. Remote commands remain uneffected, as I was still able to start remote conditioning on my vehicle via the MyBMW app, despite the status refresh failing/being stale.

The car is a 2019 G05, running 07/2021.85. I rarely use the BMW connected app, and there are no other intergations setup against this BMW account. HA is on a RPi 4, running the latest version:

Home Assistant 2022.9.5
Supervisor 2022.09.1
Operating System 9.0
Frontend 20220907.2 - latest

Please let me know if there's additional logging / information I can provide to assist. Thanks in advance! (It's great work BTW)

2022-09-19 11:41:52.341 ERROR (MainThread) [homeassistant.components.bmw_connected_drive.coordinator] Error fetching bmw_connected_drive-EMAIL_REDACTED data: Error communicating with BMW API: Client error '403 Quota Exceeded' for url 'https://cocoapi.bmwgroup.us/eadrax-vcs/v2/vehicles'
2022-09-19 11:56:59.529 ERROR (MainThread) [homeassistant.components.bmw_connected_drive.coordinator] Error fetching bmw_connected_drive-EMAIL_REDACTED data: Error communicating with BMW API: Client error '403 Quota Exceeded' for url 'https://cocoapi.bmwgroup.us/eadrax-vcs/v2/vehicles/<REDACTED>/state'
2022-09-19 12:12:01.467 ERROR (MainThread) [homeassistant.components.bmw_connected_drive.coordinator] Error fetching bmw_connected_drive-EMAIL_REDACTED data: Error communicating with BMW API: Client error '403 Quota Exceeded' for url 'https://cocoapi.bmwgroup.us/eadrax-vcs/v2/vehicles'
2022-09-19 12:22:03.312 ERROR (MainThread) [homeassistant.components.bmw_connected_drive.coordinator] Error fetching bmw_connected_drive-EMAIL_REDACTED data: Error communicating with BMW API: Client error '403 Quota Exceeded' for url 'https://cocoapi.bmwgroup.us/eadrax-vcs/v2/vehicles'
2022-09-19 12:42:08.319 ERROR (MainThread) [homeassistant.components.bmw_connected_drive.coordinator] Error fetching bmw_connected_drive-EMAIL_REDACTED data: Error communicating with BMW API: Client error '403 Quota Exceeded' for url 'https://cocoapi.bmwgroup.us/eadrax-vcs/v2/vehicles'
2022-09-19 13:02:12.676 ERROR (MainThread) [homeassistant.components.bmw_connected_drive.coordinator] Error fetching bmw_connected_drive-EMAIL_REDACTED data: Error communicating with BMW API: Client error '403 Quota Exceeded' for url 'https://cocoapi.bmwgroup.us/eadrax-vcs/v2/vehicles'
2022-09-19 13:17:14.360 ERROR (MainThread) [homeassistant.components.bmw_connected_drive.coordinator] Error fetching bmw_connected_drive-EMAIL_REDACTED data: Error communicating with BMW API: Client error '403 Quota Exceeded' for url 'https://cocoapi.bmwgroup.us/eadrax-vcs/v2/vehicles'
2022-09-19 14:02:18.469 ERROR (MainThread) [homeassistant.components.bmw_connected_drive.coordinator] Error fetching bmw_connected_drive-EMAIL_REDACTED data: Error communicating with BMW API: Client error '403 Quota Exceeded' for url 'https://cocoapi.bmwgroup.us/eadrax-vcs/v2/vehicles'
2022-09-19 14:17:19.415 ERROR (MainThread) [homeassistant.components.bmw_connected_drive.coordinator] Error fetching bmw_connected_drive-EMAIL_REDACTED data: Error communicating with BMW API: Client error '403 Quota Exceeded' for url 'https://cocoapi.bmwgroup.us/eadrax-vcs/v2/vehicles'
# I disabled 'Polling' in the HA BMW Connectect integration under "system settings". This caused the same error, but from a different call site in the code:
2022-09-19 14:55:04.557 WARNING (MainThread) [homeassistant.config_entries] Config entry 'EMAIL_REDACTED' for bmw_connected_drive integration not ready yet: Error communicating with BMW API: Client error '403 Quota Exceeded' for url 'https://cocoapi.bmwgroup.us/eadrax-vcs/v2/vehicles'

Expected behavior

Near 100% availability of BMW connected entities in HA and official BMW Connected app by staying within allocated API quota.

Which Home Assistant version are you using?

2022.9.5

What was the last working version of Home Assistant Core?

No response

What is your region?

North America

MyBMW website

Number of cars

Output of bimmer_connected fingerprint

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

MyForest commented 1 year ago

This may help quickly prove it's the line you are expecting:

DEBUG:asyncio:Using selector: EpollSelector
DEBUG:bimmer_connected.account:Getting vehicle list
DEBUG:bimmer_connected.api.authentication:Authenticating with MyBMW flow for North America & Rest of World.
DEBUG:httpx._client:HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-ucs/v1/presentation/oauth/config "HTTP/1.1 200 OK"
DEBUG:httpx._client:HTTP Request: POST https://customer.bmwgroup.com/gcdm/oauth/authenticate "HTTP/1.1 200 OK"
DEBUG:httpx._client:HTTP Request: POST https://customer.bmwgroup.com/gcdm/oauth/authenticate "HTTP/1.1 302 Found"
DEBUG:httpx._client:HTTP Request: POST https://customer.bmwgroup.com/gcdm/oauth/token "HTTP/1.1 200 OK"
DEBUG:httpx._client:HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles "HTTP/1.1 403 Quota Exceeded"
Traceback (most recent call last):
  File "/usr/local/bin/bimmerconnected", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/cli.py", line 285, in main
    loop.run_until_complete(args.func(args))
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/cli.py", line 105, in get_status
    await account.get_vehicles()
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/account.py", line 69, in get_vehicles
    vehicles_responses: List[httpx.Response] = [
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/account.py", line 70, in <listcomp>
    await client.get(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1751, in get
    return await self.request(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1700, in _send_handling_redirects
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    await hook(response)
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/api/client.py", line 66, in raise_for_status_event_handler
    response.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/httpx/_models.py", line 736, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '403 Quota Exceeded' for url 'https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles'
For more information check: https://httpstatuses.com/403

Changing const.py to:

USER_AGENT = "Dart/2.15 (dart:io)"

causes a different exception which suggests X_USER_AGENT needs to be a different value too (like this), but I'm afraid I don't have the tools to sniff the new value:

DEBUG:asyncio:Using selector: EpollSelector
DEBUG:bimmer_connected.account:Getting vehicle list
DEBUG:bimmer_connected.api.authentication:Authenticating with MyBMW flow for North America & Rest of World.
DEBUG:httpx._client:HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-ucs/v1/presentation/oauth/config "HTTP/1.1 400 Bad Request"
ERROR:bimmer_connected.api.authentication:Authentication error: {"statusCode":400,"message":"Missing or malformed x-user-agent header","error":"Bad Request"}
Traceback (most recent call last):
  File "/usr/local/bin/bimmerconnected", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/cli.py", line 285, in main
    loop.run_until_complete(args.func(args))
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/cli.py", line 105, in get_status
    await account.get_vehicles()
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/account.py", line 69, in get_vehicles
    vehicles_responses: List[httpx.Response] = [
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/account.py", line 70, in <listcomp>
    await client.get(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1751, in get
    return await self.request(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1639, in _send_handling_auth
    request = await auth_flow.__anext__()
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/api/authentication.py", line 76, in async_auth_flow
    await self.login()
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/api/authentication.py", line 111, in login
    token_data = await self._login_row_na()
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/api/authentication.py", line 201, in _login_row_na
    handle_http_status_error(ex, "Authentication", _LOGGER)
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/api/utils.py", line 52, in handle_http_status_error
    raise ex
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/api/authentication.py", line 133, in _login_row_na
    r_oauth_settings = await client.get(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1751, in get
    return await self.request(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1700, in _send_handling_redirects
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    await hook(response)
  File "/usr/local/lib/python3.10/site-packages/bimmer_connected/api/authentication.py", line 354, in raise_for_status_event_handler
    response.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/httpx/_models.py", line 736, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://cocoapi.bmwgroup.com/eadrax-ucs/v1/presentation/oauth/config'
For more information check: https://httpstatuses.com/400
MyForest commented 1 year ago

Maybe less useful, but it was fine at 2022-09-19T17:40:02+00:00 and broken at 2022-09-19T17:45:02+00:00.

rikroe commented 1 year ago

Yeah, BMW seems to play around with the quotas right now. I had the issue yesterday in rest_of_world, but it self-recovered, while still showing the error sometimes:

2022-09-20 08:06:52.526 DEBUG (MainThread) [httpx._client] HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles "HTTP/1.1 200 OK"
2022-09-20 08:06:52.901 DEBUG (MainThread) [httpx._client] HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles "HTTP/1.1 200 OK"
2022-09-20 08:06:53.454 DEBUG (MainThread) [httpx._client] HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles/VIN/state "HTTP/1.1 200 OK"
2022-09-20 08:06:53.458 DEBUG (MainThread) [custom_components.bmw_connected_drive.coordinator] Finished fetching bmw_connected_drive-ACCOUNT data in 2.457 seconds (success: True)
2022-09-20 08:11:53.893 DEBUG (MainThread) [httpx._client] HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles "HTTP/1.1 200 OK"
2022-09-20 08:11:54.309 DEBUG (MainThread) [httpx._client] HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles "HTTP/1.1 200 OK"
2022-09-20 08:11:54.358 DEBUG (MainThread) [httpx._client] HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles/VIN/state "HTTP/1.1 403 Quota Exceeded"
2022-09-20 08:11:54.361 ERROR (MainThread) [custom_components.bmw_connected_drive.coordinator] Error fetching bmw_connected_drive-ACCOUNT data: Error communicating with BMW API: Client error '403 Quota Exceeded' for url 'https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles/VIN/state'
2022-09-20 08:11:54.362 DEBUG (MainThread) [custom_components.bmw_connected_drive.coordinator] Finished fetching bmw_connected_drive-ACCOUNT data in 1.361 seconds (success: False)
2022-09-20 08:16:54.231 DEBUG (MainThread) [httpx._client] HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles "HTTP/1.1 403 Quota Exceeded"
2022-09-20 08:16:54.233 DEBUG (MainThread) [custom_components.bmw_connected_drive.coordinator] Finished fetching bmw_connected_drive-ACCOUNT data in 0.233 seconds (success: False)
2022-09-20 08:21:54.360 DEBUG (MainThread) [httpx._client] HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles "HTTP/1.1 403 Quota Exceeded"
2022-09-20 08:21:54.362 DEBUG (MainThread) [custom_components.bmw_connected_drive.coordinator] Finished fetching bmw_connected_drive-ACCOUNT data in 0.361 seconds (success: False)
2022-09-20 08:26:54.744 DEBUG (MainThread) [httpx._client] HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles "HTTP/1.1 200 OK"
2022-09-20 08:26:55.065 DEBUG (MainThread) [httpx._client] HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles "HTTP/1.1 200 OK"
2022-09-20 08:26:55.537 DEBUG (MainThread) [httpx._client] HTTP Request: GET https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles/VIN/state "HTTP/1.1 200 OK"
2022-09-20 08:26:55.541 INFO (MainThread) [custom_components.bmw_connected_drive.coordinator] Fetching bmw_connected_drive-ACCOUNT data recovered

As we do a minimum of 3 requests (one for BMW vehicle list, one for MINI vehicle list, one for each vehicle), it could be that they are sometimes too close by each other (which would be weird since the BMW app does one for BMW vehicle list and then for each vehicle, too).

Either we are missing something in our HTTP header that the MyBMW app uses to get around the Quota or they just had a config error.

Changing the user agent won't fix the problem - it has be be very specifically set otherwise you'll get the bad request.

EDIT: And it is on/off again... :D

turribeach commented 1 year ago

Getting the same error (403 Quota Exceeded). Does changing the user agent fix the issue?

rikroe commented 1 year ago

No, changing the user agent won't fix the issue.

Judging what is being reported in https://github.com/home-assistant/core/issues/78792, the MyBMW app seems to have some issues too.

Until this is working again, debugging will be hard.

turribeach commented 1 year ago

My car even dissapeared from MyBMW app! I tried adding it again but no luck...

jprouty commented 1 year ago

Sorry for posting here; I wasn't sure where made sense (here or HA core).

It seems that the core component has the scan/update interval here. It ought to be possible for all users to configure that interval if needed via the frontend (probably a FR for the core component).

In the interim, I am going to manually reduce the scan interval to 900 (or every 15minutes) and see if the issue disappears (ie, get within quota).

For anyone else that wants to try (requires advanced level), you can: 1) (depending on which type of installation) you'll need to install root ssh 1) ssh root@homeassistant.local -p 22222 1) Attach to your running docker core instance: docker ps | grep homeassistant and docker exec -it 3bcd52da44ef /bin/bash (3bcd52da44ef is my instance id for example). 1) Edit: vi /usr/src/homeassistant/homeassistant/components/bmw_connected_drive/coordinator.py to change 300 to whatever value you'd like to try. 1) ha core restart from the root shell (not core shell via docker).

danhusan commented 1 year ago

My car even dissapeared from MyBMW app! I tried adding it again but no luck...

Had the same thing happen yesterday. Over night it came back by itself. I´ve since then disabled the connected drive integration in HA, but I am struggling with quota issues today anyhow. App will happily send commands to the car, but SOC will not update. Re-logging in the APP once again yields 0 cars.

zoobuv101 commented 1 year ago

I can confirm the same, some time between last friday and today, the inegration has stopped with the error "Retrying setup: Error communicating with BMW API: Client error '403 Quota Exceeded' for url 'https://cocoapi.bmwgroup.com/eadrax-vcs/v2/vehicles' For more..."

The car is also missing from the app. When I re-added the chassis number the app confirmed that the car was already attached to the account that that it should appear soon.

turribeach commented 1 year ago

Does anyone know what the actual quota is? Why can't BMW communicate with their customers? It really makes me want to buy a Tesla!

jprouty commented 1 year ago

@turribeach I just left an educated guess over in the HA core issue.

zoobuv101 commented 1 year ago

From https://bimmerist.com/bmw-connecteddrive-not-working-full-guide/

_ConnectedDrive Discontinuation for 3G [UPDATED] Mobile carriers decided to phase out 3G networks throughout 2022 to make room for newer technologies. This decision affected (and angered) many BMW owners whose ConnectedDrive services stopped working. Although the shutting down of 3G services was beyond BMW’s control, they stopped the renewal of ConnectedDrive contracts already in 2021.

Basically, this means no more ConnectedDrive services for cars equipped with TCB-3G, COMBOX-NG, and ATM-3G telematics control units. In general, this includes BMWs from the model year 2016 and older._

I guess no more connected drive then

turribeach commented 1 year ago

My car is a 2019 X1 so I am pretty sure it is 4G already.

turribeach commented 1 year ago

@turribeach I just left an educated guess over in the HA core issue.

Left it just over 5hs alone and tried again, same error. No looking good. Let's hope it's a 24hs ban, not permanent...

turribeach commented 1 year ago

@turribeach I just left an educated guess over in the HA core issue.

Left it just over 5hs alone and tried again, same error. No looking good. Let's hope it's a 24hs ban, not permanent...

Actually I just read your message properly, so it looks like it should reset at midnight UTC, I will give it a try if I am still up otherwise I will try tomorrow.

turribeach commented 1 year ago

Confirmed it resets at midnight UTC! Also the app now shows my car as well. I checked my logs and I got banned after around 6hs of doing calls every 5mins so I have increased my frequency to 20 mins which get me to 24hs without being banned.

jprouty commented 1 year ago

My requests are going through as well, right on cue. That's the first half of the theory: let's see if a reduction in the scan interval gets us through the day.

rikroe commented 1 year ago

Thanks for the debugging guys! I'll implement a higher polling interval (probably 30 minutes to be safe) later today and make it configurable via the integration options.

nottledim commented 1 year ago

My MyBMW app doesn't work. I thought I'd log-out and log-in again. Big mistake; it doesn't even know I exist. Eventually I did something and it reported connection issues.

It seem like BMW's system is borked. I guess we just have to be patient.

rikroe commented 1 year ago

It is probably related - from what I gathered from the comments here and in https://github.com/home-assistant/core/issues/78792, there seems to be a daily quota for the MyBMW API per account. Meaning if your account has reached the quota due to the constant polling from HA, your account will also be blocked in the the MyBMW app.

It probably resets regularly at 0:00 UTC, so by then you should be able to log back into the MyBMW app and get your cars back (provided you have disabled polling in HA not to block you again directly).

MiaoWaSeed commented 1 year ago

Even when ha integration says 403 and main screen of MyBMW app not working. The state of charge (i3) in the app still updates without problem. So assume that’s via a different api? And separate quota per api?

jprouty commented 1 year ago

Quick update: I have been green for 15 hrs straight now, with the reduced poll interval of 15m (900). @rikroe I like your suggestion to reduce/cache the 2x vehicle API calls. If you are looking for an example of another integration that does this, the Mitsubishi Kumo HACS integration has a prefer_cache setting that does something similar.

@MiaoWaSeed That was my experience too (ie, I could still remote start the vehicle from MyBMW). My intuition would be that the quota is on a per-endpoint basis (ie, the get vehicles endpoint is exceeding quota).

nottledim commented 1 year ago

Could I suggest that the poll rate be different for when the car is charging from when it's not. Relevant to EV only I suppose. One of my main uses of bimmer/HA is to tell me when to get the car from the charge point. There could be an update button rather than dynamically changing poll rate. Perhaps the poll rate should decrease if there are no status changes. No point in polling when the car is parked-up. It could be woken up again with the update button. Or some such. Just a thought.

BTW I reported the MyApp failure to BMW. The technical team are going to contact me within 2 to 6 weeks. That's what I call service. :-)

rikroe commented 1 year ago

Could I suggest that the poll rate be different for when the car is charging from when it's not.

That is something you can do already. Disable automatic polling in the integration's system options and then create one or multiple automations that trigger the homeassistant.update_entity for one of the entities at your convenience.

This is also the best workaround for now as changing an otherwise adjustable interval is not desired by the HA team.

jprouty commented 1 year ago

@rikroe, might I suggest we close this issue in favor of the HA core issue (to keep the conversation/updates in one place)? Feel free to reopen if you feel otherwise.

robsonke commented 1 year ago

Fyi, I expect that bmw will come up with a solution too. For example Tibber also isn't able to control the charging anymore

github-actions[bot] commented 1 year ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.