rework authentication and API structure after VW has switched off CarNet

[robinostlund]

@oliverrahner i let you decide when you feel this is ready for release, i believe it may be better to release it once when it is sort of working as i guess people will find missing things as there are different values depending on which type of car you have 😄

[den4uk]

@oliverrahner i let you decide when you feel this is ready for release, i believe it may be better to release it once when it is sort of working as i guess people will find missing things as there are different values depending on which type of car you have 😄

Could you do a pre-release, so it would be possible to install it via HACS, and tested by wider audiences?

[oliverrahner]

Thanks @robinostlund I currently have access to a diesel, a hybrid and an electric car, and I’m also trying to cater for some things like missing glass roof and the like. I tend to go for one release once I have „all“ entities readable, and another one for actually controlling settings.

[oliverrahner]

@oliverrahner i let you decide when you feel this is ready for release, i believe it may be better to release it once when it is sort of working as i guess people will find missing things as there are different values depending on which type of car you have 😄

Could you do a pre-release, so it would be possible to install it via HACS, and tested by wider audiences?

Unfortunately, it’s not the integration I’m reworking, but the underlying library. I don’t have a creative idea right now how we could make these „alphas“ available for preview… do you, @robinostlund?

[den4uk]

@oliverrahner i let you decide when you feel this is ready for release, i believe it may be better to release it once when it is sort of working as i guess people will find missing things as there are different values depending on which type of car you have 😄

Could you do a pre-release, so it would be possible to install it via HACS, and tested by wider audiences?

Unfortunately, it’s not the integration I’m reworking, but the underlying library. I don’t have a creative idea right now how we could make these „alphas“ available for preview… do you, @robinostlund?

The owner (@robinostlund) could take a tag of your branch, and create a pre-release. I would test it on my ID.3 tonight.

[TheBlackBGD]

I would test it also.

[LucTs]

Mee too, I do have a 2022 Touran.

[robinostlund]

Is it ok if i do it tomorrow morning? Dont have any computer infront of me now 😊

@oliverrahner i think you should have access to do this also now when you are a collaborator 😊

[TheBlackBGD]

Let us know when we can update from hacs ill do it roght away, for me is also ok for tomorrow as i see @oliverrahner used all my apicalls and the vw application is frozen 😂😂😂 No worries :)

[virtualdj]

I don’t have a creative idea right now how we could make these „alphas“ available for preview…

Actually if one runs the Python code in the example with the library saved as a subfolder ("download as ZIP") and with the correct login information, we/you could get information about the sensors that are working or not.

It is not the easy of use of Home Assistant sensors, but for debugging a lot of configurations it could be a viable solution.

[oliverrahner]

Let us know when we can update from hacs ill do it roght away, for me is also ok for tomorrow as i see @oliverrahner used all my apicalls and the vw application is frozen 😂😂😂 No worries :)

That is very strange... I tested only very little with your car and mostly with mine and the one from @michiii1337, and ours still work... But I can confirm that I also get HTTP 429s from the VW API for your credentials.

We need to monitor that, because I have set up all our 3 cars in my Dev HA environment, so it might be some case that produces more API calls for electric cars than for others.

[TheBlackBGD]

Let us know when we can update from hacs ill do it roght away, for me is also ok for tomorrow as i see @oliverrahner used all my apicalls and the vw application is frozen 😂😂😂 No worries :)

That is very strange... I tested only very little with your car and mostly with mine and the one from @michiii1337, and ours still work... But I can confirm that I also get HTTP 429s from the VW API for your credentials.

We need to monitor that, because I have set up all our 3 cars in my Dev HA environment, so it might be some case that produces more API calls for electric cars than for others.

@oliverrahner it is possible i dont know i use the app also almoast every time i go somewhere to heat the car, and i have also the other integration installed. But to have some ideeas i will delete the other integration and see from tomorrow what is hapening, ok?

[oliverrahner]

@oliverrahner it is possible i dont know i use the app also almoast every time i go somewhere to heat the car, and i have also the other integration installed. But to have some ideeas i will delete the other integration and see from tomorrow what is hapening, ok?

Sounds good! I also removed it now from my Dev instances, so we can investigate better.

[TheBlackBGD]

@oliverrahner it is possible i dont know i use the app also almoast every time i go somewhere to heat the car, and i have also the other integration installed. But to have some ideeas i will delete the other integration and see from tomorrow what is hapening, ok?

Sounds good! I also removed it now from my Dev instances, so we can investigate better.

@oliverrahner I have deleted the other integration. As it happened last time after midnight it worked again, so from now it will be my mobile app and you for the api calls. If i can help with something let me know!

[virtualdj]

@oliverrahner I've captured my self.attrs in the attachment, if you need further data to model the properties. I've changed the sensitive data of course, such as location and VIN.

data.txt

[iainfogg]

Hi @oliverrahner the work I'd been doing was all experimental so far, trying to compare (by running both in debuggers) the login process from this library with the WeConnect-python library, but it looks like you've got the login stuff working already, which is great!

I've been working in VS Code using GitHub Codespaces so far, which is ideal for working on a single repo - I wanted to check what setup you're using for running this within HA while doing dev on it, so I can get the same setup and see if I can contribute something.

[TheBlackBGD]

@oliverrahner i saw the beta version was released, my question is can i install it and login, will it interfere in your tests with my account ? If yes i will wait stil until you tell me. Let me know please

[oliverrahner]

I've been working in VS Code using GitHub Codespaces so far, which is ideal for working on a single repo - I wanted to check what setup you're using for running this within HA while doing dev on it, so I can get the same setup and see if I can contribute something.

What I'm doing is a bit hacky, but works:

• run HomeAssistant in a local docker container, mapping my local dev folder into the container at /config/custom_components/volkswagencarnet/volkswagencarnet
• install my local dev variant like this pip install -e /config/custom_components/volkswagencarnet/volkswagencarnet
• install Volkswagen integration
• modify manifest.json for the integration and remove the fixed version from the reference to volkswagencarnet

When I modify anything, I tend to remove the __pycache__ folder and then restart HomeAssistant, even though both shouldn't be necessary

[stickpin]

@oliverrahner I've created a PR to fix the fuel sensor for combustion cars like Touran: https://github.com/robinostlund/volkswagencarnet/pull/217

[oliverrahner]

Thanks @stickpin, but I already fixed it in https://github.com/robinostlund/volkswagencarnet/pull/216/commits/13d5df144b77075004436e636915a18e97532f77 :)

[stickpin]

Thanks @stickpin, but I already fixed it in 13d5df1 :)

Not really, please see my commit. :)

[oliverrahner]

@robinostlund I believe we could have another beta release now.

Just to sum up the current development status:

added:

• locked entities
• parking_position
• last_trip information

[oliverrahner]

Thanks @stickpin, but I already fixed it in 13d5df1 :)

Not really, please see my commit. :)

Didn't look deep enough 😅 Sorry, and thanks!

[robinostlund]

@robinostlund I believe we could have another beta release now.

Just to sum up the current development status:

added:

• locked entities
• parking_position
• last_trip information

New beta version has now been released 😄 Thank you ❤️

[TheBlackBGD]

Excelent work guys you really are awsome !!!! Updated ! Coffe to go !!!!

[robinostlund]

@oliverrahner what do you say about merging this pr to master and then starting with new ones for the missing parts? :)

[oliverrahner]

Totally agreed. My suggestion is to keep this PR open for some time (a few days?) so we know that it doesn't introduce new issues, and then merge it. I will commit my further efforts to separate branch based off of this one so we have a clear distinction.

[iainfogg]

I've been working in VS Code using GitHub Codespaces so far, which is ideal for working on a single repo - I wanted to check what setup you're using for running this within HA while doing dev on it, so I can get the same setup and see if I can contribute something.

What I'm doing is a bit hacky, but works:

• run HomeAssistant in a local docker container, mapping my local dev folder into the container at /config/custom_components/volkswagencarnet/volkswagencarnet
• install my local dev variant like this pip install -e /config/custom_components/volkswagencarnet/volkswagencarnet
• install Volkswagen integration
• modify manifest.json for the integration and remove the fixed version from the reference to volkswagencarnet

When I modify anything, I tend to remove the __pycache__ folder and then restart HomeAssistant, even though both shouldn't be necessary

Thanks, hacky beats not working, which is where I got to last night!

So you're running the dev version of volkswagencarnet and the standard homeassistant-volkswagencarnet (plus the modification in the manifest file). I was trying to get dev versions of both, so that's simplified it.

I'll try and get that up and running.

[stickpin]

@oliverrahner another small fix for the Parking Time sensor: https://github.com/robinostlund/volkswagencarnet/pull/218 and one more to remove duplicate entries: https://github.com/robinostlund/volkswagencarnet/pull/219

[iainfogg]

OK, got my IDE set up but as I'm running off your branch right now I can't commit, I'll sort that later, but for now here's a contribution:

vw-vehicle.py:

• Line 1096 - change
  • From park_time_utc: datetime = self.attrs.get("findCarResponse", {}).get("parkingTimeUTC", "Unknown")
  • To park_time_utc: datetime = find_path(self.attrs, "parkingposition.carCapturedTimestamp")
• Line 1108 - change
  • From return "parkingTimeUTC" in self.attrs.get("findCarResponse", {})
  • To return find_path(self.attrs, "parkingposition.carCapturedTimestamp")

I'll try and do others on my own fork, or you can make me a contributor and I'll push others to your branch, whatever you prefer @oliverrahner

[stickpin]

@iainfogg it's already solved in this PR https://github.com/robinostlund/volkswagencarnet/pull/218. It's waiting for review and merge. :)

[iainfogg]

@iainfogg it's already solved in this PR https://github.com/robinostlund/volkswagencarnet/pull/218. It's waiting for review and merge. :)

Ha, ok thanks for letting me know, hadn't spotted that 🤦

Might be good if we can come up with a way of flagging up which properties we're working on, so we don't try and solve the same thing.

[stickpin]

@oliverrahner it's only me or did VW set the API rate limit to such a low value that it becomes completely useless? In the last two days, I have been getting rate-limited very fast, without doing much. How does it behave for you?

[oliverrahner]

@stickpin I have actually turned off all integrations right now after hearing so many problems from the other folks, because I can't work on anything when my available accounts are blocked. I'm trying to get a grip on that issue currently, by analyzing what the actual limit is and what contributes towards it. E.g., does one call to /vehicle/v1/vehicles/{vin}/selectivestatus count as one request per job requested? Having some numbers on this would also mean that we could rather easily spot any differences in VWs API in the future.

[oliverrahner]

Initial thoughts on this: Right now, we are requesting 13 endpoints per update, if every selectivestatus job counts on its own. I have already locally removed 2 endpoints that make no sense. Some interesting endpoints (like the legacy parking heater) are still missing. That number might already be an issue in itself, but I don't have a good idea right now how to handle this. Apart from the thoughts some users had to introduce an On/Off switch that controls regular polling (https://github.com/robinostlund/homeassistant-volkswagencarnet/issues/511#issuecomment-1837104461). I don't really like this idea, but if the limits are too low to be useful, then exposing the level of control to the user might be necessary.

[oliverrahner]

I now try to make some rough counts using this script: https://gist.github.com/oliverrahner/4c9b616b1f934ac7df9ec423b78723dc (designed to be called from a local clone of the library, because of the reference to tests.credentials)

With that setup (only calling the access endpoint), I currently have ~400 calls and it still works...

[stickpin]

@oliverrahner btw, I think not all selectivestatus jobs are required for all the cars, so it might make sense to call only relevant jobs for the specific car model after initial setup.

Currently, I think we have those jobs:

  "access",
  "fuelStatus",
  "honkAndFlash",
  "userCapabilities",
  "vehicleLights",
  "vehicleHealthInspection",
  "measurements",
  "charging",
  "climatisation",
  "automation",

For example, I think those are not relevant for my car:

 "charging",
  "climatisation",
  "automation",
  "honkAndFlash",

The only problem is to do the mapping. If I am not mistaken capabilities expose the list of the relevant jobs for the car model, or am I missing something? Maybe we should have some sort of local white list that will compare it with capabilities list?

[oliverrahner]

We are already doing this (and the old version of this lib did it thoroughly):

    async def get_parkingposition(self):
        """Fetch parking position if supported."""
        if self._services.get("parkingPosition", {}).get("active", False):

It's just a matter of either splitting up the get_selectivestatus calls into one method for each job or checking each passed job entry against the capabilities. I prefer the latter for performance reasons.

Also, the honkAndFlash service doesn't return any data (but might still contribute to the limit) and the userCapabilities are not used for anything on our side, so I already removed them.

[iainfogg]

I've been running the integration every 120 minutes and it's been ok at that. I've also emailed Volkswagen support and asked about what the limit is, but I doubt they'll tell me.

I'd guess all API calls on a login count towards the limit, but it's hard / impossible to know.

I'd been wondering about the bit in the integration config UI that asks what items you want to keep updated being sorted so it was grouped by what's covered in each API call, as a way of potentially cutting down calls, by people being able to avoid having too many for less important entities.

Also, I'd wondered how much stuff could be cached - about what the car supports, about entities that don't change (eg battery capacity), or entities that need updating much less often eg cause to next service.

[oliverrahner]

As long as static entities are in the same API call as dynamic ones, the caching approach wouldn't save anything :( Combining the config info from the integration with the actual requests from the library would be possible, but I really want to understand the way the rate limiting works a bit better first.

I managed to do > 1.700 API calls yesterday without being blocked, while calling only one service with each call, which would be enough for one update every minute over a whole day... I'll try to hit the limit today, to find out what the actual limit is.

[oliverrahner]

I have been running requests to https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVWZZZ3xxx/selectivestatus?jobs=access almost all day, with a 1 second delay, starting this morning, and it doesn't 429 me... I'll start calling multiple jobs now.

[stickpin]

I have been running requests to https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVWZZZ3xxx/selectivestatus?jobs=access almost all day, with a 1 second delay, starting this morning, and it doesn't 429 me... I'll start calling multiple jobs now.

I made some changes as well, now I have 4 requests every 15 minutes. I am sending those 4 jobs every time. Let's see if and when I will get blocked. Started at 15:00CET. jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements

[stickpin]

@oliverrahner just thinking out loud... VW servers might react not to the amount of requests but to the request per/second. Currently, all 4 requests request are being sent almost simultaneously.

If I get blocked today, I will try to add a 1-second delay between each API request.

[stickpin]

@oliverrahner there is also something strange going on during the token renewal. The same 4 API requests are being called 4 times.

HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements"
HTTP GET "https://emea.bff.cariad.digital/vehicle/v2/vehicles"
HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition"
HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last"

Here is the full log:

2023-12-04 16:00:56.982 DEBUG (MainThread) [volkswagencarnet.vw_connection] Tokens have expired. Try to fetch new tokens.
2023-12-04 16:00:57.188 WARNING (MainThread) [volkswagencarnet.vw_connection] Something went wrong when refreshing VW account tokens.
2023-12-04 16:00:57.189 INFO (MainThread) [volkswagencarnet.vw_connection] Session expired. Initiating new login for VW account.
2023-12-04 16:00:57.189 DEBUG (MainThread) [volkswagencarnet.vw_connection] Initiating new login
2023-12-04 16:00:57.511 DEBUG (MainThread) [volkswagencarnet.vw_connection] Got authorization endpoint
2023-12-04 16:00:57.703 DEBUG (MainThread) [volkswagencarnet.vw_connection] Authenticating with email and password.
2023-12-04 16:00:58.035 DEBUG (MainThread) [volkswagencarnet.vw_connection] Parsing login response.
2023-12-04 16:00:58.265 DEBUG (MainThread) [volkswagencarnet.vw_connection] Login successful, received authorization code.
2023-12-04 16:00:58.265 DEBUG (MainThread) [volkswagencarnet.vw_connection] Trying to fetch user identity tokens.
2023-12-04 16:00:58.445 DEBUG (MainThread) [volkswagencarnet.vw_connection] User identity token verified OK.
2023-12-04 16:00:58.445 INFO (MainThread) [volkswagencarnet.vw_connection] Successfully logged in
2023-12-04 16:00:58.446 DEBUG (MainThread) [volkswagencarnet.vw_connection] Fetching vehicles associated with account
2023-12-04 16:00:58.446 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v2/vehicles"
2023-12-04 16:00:59.526 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v2/vehicles" returned with status code [200]
2023-12-04 16:00:59.526 DEBUG (MainThread) [volkswagencarnet.vw_connection] Found vehicle(s) associated with account.
2023-12-04 16:00:59.526 DEBUG (MainThread) [volkswagencarnet.vw_connection] Going to call vehicle updates
2023-12-04 16:00:59.527 DEBUG (MainThread) [volkswagencarnet.vw_vehicle] Attempting discovery of supported API endpoints for vehicle.
2023-12-04 16:00:59.527 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/capabilities"
2023-12-04 16:00:59.530 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements"
2023-12-04 16:00:59.531 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v2/vehicles"
2023-12-04 16:00:59.532 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition"
2023-12-04 16:00:59.533 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last"
2023-12-04 16:00:59.978 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/capabilities" returned with status code [200]
2023-12-04 16:00:59.978 DEBUG (MainThread) [volkswagencarnet.vw_vehicle] Discovered enabled service: measurements
2023-12-04 16:00:59.978 DEBUG (MainThread) [volkswagencarnet.vw_vehicle] Service: access is disabled because of reason: ['MissingLicense', 'LicenseExpired']
2023-12-04 16:00:59.978 DEBUG (MainThread) [volkswagencarnet.vw_vehicle] Discovered enabled service: parkingPosition
2023-12-04 16:00:59.978 DEBUG (MainThread) [volkswagencarnet.vw_vehicle] Discovered enabled service: tripStatistics
2023-12-04 16:00:59.978 DEBUG (MainThread) [volkswagencarnet.vw_vehicle] API endpoints: {'access': {'active': False}, 'tripStatistics': {'active': True, 'expiration': datetime.datetime(2051, 4, 12, 0, 0, tzinfo=datetime.timezone.utc), 'operations': ['deleteTripdataCyclic', 'getTripdataCyclic', 'deleteTripdataLongterm', 'getTripdataLongterm', 'deleteTripdataShortterm', 'deleteTripdataShorttermByID', 'getTripdataCyclicLast', 'deleteTripdataCyclicByID', 'getTripdataLongtermLast', 'deleteTripdataLongtermByID', 'getTripdataShortterm', 'getTripdataShorttermLast']}, 'measurements': {'active': True, 'expiration': datetime.datetime(2051, 4, 12, 0, 0, tzinfo=datetime.timezone.utc), 'operations': ['getMeasurements']}, 'parkingPosition': {'active': True, 'expiration': datetime.datetime(2051, 4, 12, 0, 0, tzinfo=datetime.timezone.utc), 'operations': ['getParkingposition']}}
2023-12-04 16:00:59.979 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements"
2023-12-04 16:00:59.983 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v2/vehicles"
2023-12-04 16:00:59.984 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition"
2023-12-04 16:00:59.985 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last"
2023-12-04 16:01:00.206 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v2/vehicles" returned with status code [200]
2023-12-04 16:01:00.227 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last" returned with status code [200]
2023-12-04 16:01:00.262 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition" returned with status code [200]
2023-12-04 16:01:00.719 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v2/vehicles" returned with status code [200]
2023-12-04 16:01:02.118 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements" returned with status code [207]
2023-12-04 16:01:02.546 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition" returned with status code [200]
2023-12-04 16:01:03.792 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last" returned with status code [200]
2023-12-04 16:01:04.342 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements" returned with status code [207]
2023-12-04 16:01:04.342 DEBUG (MainThread) [volkswagencarnet.vw_connection] Going to call vehicle updates
2023-12-04 16:01:04.343 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements"
2023-12-04 16:01:04.348 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v2/vehicles"
2023-12-04 16:01:04.350 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition"
2023-12-04 16:01:04.353 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last"
2023-12-04 16:01:04.355 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements"
2023-12-04 16:01:04.356 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v2/vehicles"
2023-12-04 16:01:04.358 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition"
2023-12-04 16:01:04.359 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last"
2023-12-04 16:01:04.734 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last" returned with status code [200]
2023-12-04 16:01:04.758 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last" returned with status code [200]
2023-12-04 16:01:04.955 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v2/vehicles" returned with status code [200]
2023-12-04 16:01:04.986 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition" returned with status code [200]
2023-12-04 16:01:05.065 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v2/vehicles" returned with status code [200]
2023-12-04 16:01:05.191 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition" returned with status code [200]
2023-12-04 16:01:05.563 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements" returned with status code [207]

[stickpin]

@oliverrahner even stranger now... after the first token refresh next refresh after 15 minutes triggered twice:

2023-12-04 16:16:07.986 DEBUG (MainThread) [volkswagencarnet.vw_connection] Going to call vehicle updates
2023-12-04 16:16:07.986 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements"
2023-12-04 16:16:07.988 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v2/vehicles"
2023-12-04 16:16:07.989 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition"
2023-12-04 16:16:07.989 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last"
2023-12-04 16:16:07.991 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements"
2023-12-04 16:16:07.991 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v2/vehicles"
2023-12-04 16:16:07.992 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition"
2023-12-04 16:16:07.993 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last"
2023-12-04 16:16:08.809 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last" returned with status code [200]
2023-12-04 16:16:08.882 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last" returned with status code [200]
2023-12-04 16:16:08.888 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition" returned with status code [200]
2023-12-04 16:16:08.894 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition" returned with status code [200]
2023-12-04 16:16:09.307 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v2/vehicles" returned with status code [200]
2023-12-04 16:16:09.318 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v2/vehicles" returned with status code [200]
2023-12-04 16:16:10.838 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements" returned with status code [207]
2023-12-04 16:16:12.804 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements" returned with status code [207]

same story repeats 30 minutes after:

2023-12-04 16:31:12.979 DEBUG (MainThread) [volkswagencarnet.vw_connection] Going to call vehicle updates
2023-12-04 16:31:12.980 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements"
2023-12-04 16:31:12.980 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v2/vehicles"
2023-12-04 16:31:12.982 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition"
2023-12-04 16:31:12.983 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last"
2023-12-04 16:31:12.983 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements"
2023-12-04 16:31:12.984 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v2/vehicles"
2023-12-04 16:31:12.984 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition"
2023-12-04 16:31:12.984 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last"
2023-12-04 16:31:13.634 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last" returned with status code [200]
2023-12-04 16:31:13.646 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition" returned with status code [200]
2023-12-04 16:31:13.850 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition" returned with status code [200]
2023-12-04 16:31:13.885 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last" returned with status code [200]
2023-12-04 16:31:13.949 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements" returned with status code [207]
2023-12-04 16:31:14.590 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements" returned with status code [207]

It might explain why we are reaching rate limit. :)

[oliverrahner]

That does look strange, I agree... (as far as I know) I haven't changed the inner logic of the lib, so I'll have to investigate why that might be happening... But also using my script above something is off with the validate_tokens call.

So far, I have had > 1.500 calls to selectivestatus, with all 8 services activated, and not getting blocked...

[stickpin]

@oliverrahner I've changed the update function to include a 1-second delay. Now it looks like this:

    async def update(self):
        """Try to fetch data for all known API endpoints."""
        if not self._discovered:
            await self.discover()
        if not self.deactivated:
            await asyncio.gather(
                # TODO: we don't check against capabilities currently, but this also doesn't seem to be neccesary
                # to be checked if we should still do it for UI purposes
                self.get_selectivestatus(
                    [
                        "access",
                        "fuelStatus",
                        #"honkAndFlash",
                        #"userCapabilities",
                        "vehicleLights",
                        "vehicleHealthInspection",
                        "measurements",
                        #"charging",
                        #"climatisation",
                        #"automation",
                    ]
                ),

                #self.get_vehicle(),
                #self.get_parkingposition(),
                #self.get_trip_last()
                #     self.get_preheater(),
                #     self.get_climater(),
                #     self.get_trip_statistic(),
                #     self.get_position(),
                #     self.get_statusreport(),
                #     self.get_charger(),
                #     self.get_timerprogramming(),
                #     return_exceptions=True,
            )
            await asyncio.sleep(1)
            await asyncio.gather(
                self.get_vehicle(),
            )
            await asyncio.sleep(1)
            await asyncio.gather(
                self.get_parkingposition(),
            )
            await asyncio.sleep(1)
            await asyncio.gather(
                self.get_trip_last()
            )
        else:
            _LOGGER.info(f"Vehicle with VIN {self.vin} is deactivated.")

And it's doing an update every 5 minutes. So far so good. Let's see what will happen after the token refresh.

[stickpin]

@oliverrahner after adding a 1-second delay, the first refresh token led to 2 duplicate requests instead of 4.

2023-12-04 17:52:40.796 DEBUG (MainThread) [volkswagencarnet.vw_connection] Going to call vehicle updates
2023-12-04 17:52:40.797 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements"
2023-12-04 17:52:40.798 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements"
2023-12-04 17:52:42.800 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements" returned with status code [207]
2023-12-04 17:52:43.117 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/selectivestatus?jobs=access,fuelStatus,vehicleLights,vehicleHealthInspection,measurements" returned with status code [207]
2023-12-04 17:52:43.802 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v2/vehicles"
2023-12-04 17:52:44.122 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v2/vehicles"
2023-12-04 17:52:44.629 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v2/vehicles" returned with status code [200]
2023-12-04 17:52:45.001 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v2/vehicles" returned with status code [200]
2023-12-04 17:52:45.644 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition"
2023-12-04 17:52:46.001 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition" returned with status code [200]
2023-12-04 17:52:46.002 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition"
2023-12-04 17:52:46.400 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/vehicles/WVXXXXX/parkingposition" returned with status code [200]
2023-12-04 17:52:47.003 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last"
2023-12-04 17:52:47.406 DEBUG (MainThread) [volkswagencarnet.vw_connection] HTTP GET "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last"
2023-12-04 17:52:47.580 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last" returned with status code [200]
2023-12-04 17:52:47.814 DEBUG (MainThread) [volkswagencarnet.vw_connection] Request for "https://emea.bff.cariad.digital/vehicle/v1/trips/WVXXXXX/shortterm/last" returned with status code [200]
2023-12-04 17:52:47.815 DEBUG (MainThread) [custom_components.volkswagencarnet] Updating data from volkswagen WeConnect

We have some race conditions on token refresh. I will try to find some time tomorrow to debug it.

[stickpin]

@oliverrahner btw... the refresh token always fails for me. Which might be the reason for this race condition.

2023-12-04 17:42:01.797 DEBUG (MainThread) [volkswagencarnet.vw_connection] Tokens about to expire. Try to fetch new tokens.
2023-12-04 17:42:02.059 WARNING (MainThread) [volkswagencarnet.vw_connection] Something went wrong when refreshing VW account tokens.
2023-12-04 17:42:02.061 INFO (MainThread) [volkswagencarnet.vw_connection] Session expired. Initiating new login for VW account.
2023-12-04 17:42:02.061 DEBUG (MainThread) [volkswagencarnet.vw_connection] Initiating new login
2023-12-04 17:42:02.485 DEBUG (MainThread) [volkswagencarnet.vw_connection] Got authorization endpoint
2023-12-04 17:42:02.659 DEBUG (MainThread) [volkswagencarnet.vw_connection] Authenticating with email and password.
2023-12-04 17:42:02.953 DEBUG (MainThread) [volkswagencarnet.vw_connection] Parsing login response.
2023-12-04 17:42:03.221 DEBUG (MainThread) [volkswagencarnet.vw_connection] Login successful, received authorization code.
2023-12-04 17:42:03.221 DEBUG (MainThread) [volkswagencarnet.vw_connection] Trying to fetch user identity tokens.
2023-12-04 17:42:03.565 DEBUG (MainThread) [volkswagencarnet.vw_connection] User identity token verified OK.
2023-12-04 17:42:03.565 INFO (MainThread) [volkswagencarnet.vw_connection] Successfully logged in
2023-12-04 17:42:03.565 DEBUG (MainThread) [volkswagencarnet.vw_connection] Fetching vehicles associated with account

[stickpin]

I've changed validate_tokens function to do login instead of refresh_token.

Let's see how it will work now.