Closed uphillbattle closed 1 year ago
What country are you in? I don't see any my toyota app, only myT on apple app store--Lucian Lazar – Senior Solution ArchitectecoIT Solutions Srl@@. +40770558273www https://ecoit.solutionsOn Oct 29, 2023, at 13:31, Tom @.> wrote: Describe the bug The Toyota integration has stopped working, seems to have happened at the same time as the car was moved (by Toyota) from the «MyT Toyota» app to the «MyToyota» app. I get this error message in the logs: 2023-10-29 12:27:21.691 ERROR (MainThread) [mytoyota] Please setup Connected Services if you want live data from the car. (JTM**)
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>
Norway. I noticed that all the Toyota sensors in HA were unavailable. I opened the MyT app to see what I could see and was met by a notification that car was moved to the MyToyota app and that I should download that. After doing that and logging in, I can see the car and its data in the new MyToyota app, but the HA integration is unsuccessful - with the error message in the OP.
I can’t anymore set up the car in the old MyT app. If I try, it responds with a message saying I have to use the new MyToyota app. I’m wondering if Toyota have set up a new API and are moving cars to the new app (perhaps staged somehow - for example by country).
I have the same behavior here in Germany after my vehicle was migrated to the new MyToyota app last week: https://play.google.com/store/apps/details?id=com.toyota.oneapp.eu Since then no connection with the Home Assistant custom component is possible.
Since the custom component was only possible due to a reverse engeneering of the old Toyota app by @DurgNomis-drol, I don't know if he will find the time to do that for the new app/API again. 😢
Maybe we should all write to Toyota via different channels and ask them to finally provide us with an official API!
I just wrote an email to Toyota. Let's see if something comes back:
Official API for Toyota Connected Services? Simon **** Mi., 1. Nov. 2023 um 12:40 To: MyTapp@toyota-europe.com
Dear Toyota App Team,
the new MyToyota App is in itself a nice revision of the previous MyT App. However, Toyota seems to have also changed the API here in the background. This poses problems for developers who offer solutions based on reverse engineering of the old MyT app. They now have to start all over again. For example, the Home Assistant Toyota integration, which is used by several thousand people, is no longer functional after a migration to the new MyToyota API: https://github.com/DurgNomis-drol/ha_toyota/issues/164
As mentioned, the MyToyota app is quite nice, but for those who want to go a step further and integrate their Toyota really "smart" into the rest of their smart home ecosystem, it is unfortunately insufficient. So would it please be possible to finally provide us with an official API for Toyota Connected Services?
You would really endear yourselves to some people and really show a will to innovate.
Best regards and many thanks in advance! Simon
Hello everybody!
This is sad to see, but I am not surprised that they made a new app. I still use the old app, so it is still working for me (but this could change in the future and will probably change med decision below if this happens).
At this point in time I have no plans to work on adopting this for the new API. The reason is that my car (Toyota Aygo) doesn't have support for many features. This is also the reason that my development of this integration stalled. As it was becoming to much work to maintain many of the features that exists when I couldn't test them myself (or enjoy them). Reverse engineering is hard work and can take a long time just to get an idea of what is possible. With the currently used API, it took me a long time to understand all the quirks. But I was also lucky that had a starting point. @CM000n provided valuable feedback in the start and help a lot. If I am gonna convert this to the new API, I am not gonna add more features then what I can support with my own car. This is to make sure it will be manageable for me in the future.
If anyone would like to take on this task, you are of cause more then welcome to take any thing for this repository.
Hi @DurgNomis-drol, sorry to hear that, but totally understandable! Unfortunately, I currently also have little time and will probably not be able to take care of an adaptation to the new API.
By the way, the migration of the vehicles to the new App/API takes place without any active interactions @DurgNomis-drol. One morning when I opened the MyT app, it simply told me to use the new app. So it's quite possible that your Aygo will be migrated soon as well ;-)
@CM000n Well then I will try not to open the App 😉 or I will have a winter project on my hand again 😆 🚀
I checked my app, and my corolla has not been moved yet, but from what i can read here, this might happen within the next days.
When this happens, maybe i can provide informations, or also can help to code a library for the new api. Let us find out, what will happen.
I moved to the new MyTotota app after buying a Corolla this week. I added the HA integration and logged in with my Toyota credentials. I only get one entity: sensor.corolla_numberplate. The rest is not loaded or available.
@nielsvdc This is exactly what I see as well, with a bZ4x
I checked my app, and my corolla has not been moved yet, but from what i can read here, this might happen within the next days.
When this happens, maybe i can provide informations, or also can help to code a library for the new api. Let us find out, what will happen.
Today I took a closer look at the decompiled MyToyota .apk file. It seems that none of the previous <endpoint>.toyota-europe.com
addresses are used anymore.
Instead, there are quite a few other URLs used in the sources/oneapp/framework/network/C19794f.java
file. Unfortunately I am not a java developer, so it is still a bit difficult for me to understand what exactly is used for what.
But when I look at the console log of the customer area of toyota.de, it still seems to make api calls against the https://cpb2cs.toyota-europe.com
address to get basic car information there. Also the https://ssoms.toyota-europe.com
addresses are still used there for authentication.
So it seems that the https://cpb2cs.toyota-europe.com
address is currently still being used for basic vehicle information (This is why the number plate sensor in the custom component has continued to work so far), but the driving statistics are coming via a different API than the previous one at https://myt-agg.toyota-europe.com/cma/api
.
An interesting address that I found in the app is this one: https://ctpa-oneapi.tceu-ctp-dev.toyotaconnectedeurope.dev
This seems to be a collection of various Toyota Connected Services API endpoints. But I would need to dig into it a bit more to understand how to authenticate against it.
I haven't look in the .apk file like you have, @CM000n, but I played around with the old API a bit and noticed that the "legacy" endpoint https://myt-agg.toyota-europe.com/cma/api/vehicles/${"vin"}/remoteControl/status
still includes charge status and remaining distance (for my bZ4x):
{
"VehicleInfo": {
"AcquisitionDatetime": "2023-11-07T12:38:17Z",
"RemoteHvacInfo": {
"Temperaturelevel": 33,
"RemoteHvacMode": 0,
"RemoteHvacProhibitionSignal": 1,
"SettingTemperature": 22.0,
"BlowerStatus": 0,
"FrontDefoggerStatus": 0,
"RearDefoggerStatus": 0,
"InsideTemperature": 26
},
"ChargeInfo": {
"EvDistanceInKm": 446.1,
"GasolineTravelableDistanceUnit": 1,
"GasolineTravelableDistance": 0,
"PlugStatus": 45,
"PlugInHistory": 33,
"RemainingChargeTime": 65535,
"EvTravelableDistance": 446.1,
"EvTravelableDistanceSubtractionRate": 18,
"ChargeRemainingAmount": 100,
"SettingChangeAcceptanceStatus": 0,
"ChargeType": 15,
"ChargeWeek": 0,
"ChargeStartTime": "42:35",
"ChargeEndTime": "42:35",
"ConnectorStatus": 5,
"BatteryPowerSupplyPossibleTime": 16383,
"ChargingStatus": "chargeComplete",
"EvDistanceWithAirCoInKm": 365.8
}
},
"ReturnCode": "000000"
}
Don't know how long that will last, though.
I haven't look in the .apk file like you have, @CM000n, but I played around with the old API a bit and noticed that the "legacy" endpoint
https://myt-agg.toyota-europe.com/cma/api/vehicles/${"vin"}/remoteControl/status
still includes charge status and remaining distance (for my bZ4x):
Good finding @uphillbattle ! If I have time, I'll play around with the old API endpoint tomorrow too.
By the way: I was just about to visit our friends at Toyota Home Assistant Integration for North America to see which endpoints are used there. I was shocked to see that the Gut repo was shut down with a DMCA notification: https://github.com/widewing/toyota-na Let's hope Toyota Europe doesn't take similar steps.
Well now I got the email, but I haven't opened the app, and it still seems to work, so I believe it only moves you to the new app if you open the old one 🎉
I can confirm up to now. Mail with the notification for shutdown of the old app has arrived. Maybe i have now a weekend-project.
Hi @DurgNomis-drol and @m-dietrich,
I took another look today and I think we are dealing with two problems here:
1. The “Please setup Connected Services …” error
This is an internal error of the mytoyota
Python lib and occurs because the structure of the data returned by the API has changed. This causes a check to see whether Connected Services are activated in the mytoyota
Python lib to fail. I've just created a pull request for this with a possible fix: https://github.com/DurgNomis-drol/mytoyota/pull/240
However, this still needs to be merged and the ha_toyota
custom component needs to be adjustet so that it uses the latest version of the mytoyota
lib. This may require major adjustments as ha_toyota
is currently using an outdated version of the mytoyota
lib.
What is your opinion here @DurgNomis-drol?
2. Changed API Endpoint for vehicle's trips?
At first I thought we were dealing with a changed API endpoint here. However, after I used the above fix locally, the following minimal example including the information about the vehicle's journeys also worked:
import asyncio
from pprint import pprint
from mytoyota.client import MyT
username = "USERNAME"
password = "PASSWORD"
brand = "toyota" # or lexus
# Get supported regions, can be passed to the optional 'region' argument of MyT
# At this moment, only the 'europe' region is supported
pprint(MyT.get_supported_regions())
client = MyT(username=username, password=password, brand=brand)
async def get_information():
pprint("Logging in...")
await client.login()
pprint("Retrieving cars...")
# Returns cars registered to your account + information about each car.
cars = await client.get_vehicles()
pprint(cars)
for car in cars:
vehicle = await client.get_vehicle_status(car)
# Calling the vehicle status object directly continues to work:
pprint("Vehicle Info..")
pprint(vehicle._vehicle_info)
pprint("Connected Services..")
pprint(vehicle._connected_services)
pprint("Odometer..")
pprint(vehicle.odometer)
pprint("Status..")
pprint(vehicle._status)
pprint("Status Legacy..")
pprint(vehicle._status_legacy)
# Colling the vehicle dashboard object works after fixing the "Connected Services Enabled" Check:
pprint(vehicle.dashboard.odometer)
# Vehicle driving statistics also still seem to be working after fixing "Connected Services Enabled" Check:
data = await asyncio.gather(
*[
client.get_driving_statistics(vehicle.vin, interval="day"),
client.get_driving_statistics(vehicle.vin, interval="isoweek"),
client.get_driving_statistics(vehicle.vin),
client.get_driving_statistics(vehicle.vin, interval="year"),
]
)
pprint(data)
# Daily stats
daily_stats = await client.get_driving_statistics(vehicle.vin, interval="day")
pprint(daily_stats)
# ISO 8601 week stats
iso_weekly_stats = await client.get_driving_statistics(vehicle.vin, interval="isoweek")
pprint(iso_weekly_stats)
# Monthly stats is returned by default
monthly_stats = await client.get_driving_statistics(vehicle.vin)
pprint(monthly_stats)
# Get year to date stats.
yearly_stats = await client.get_driving_statistics(vehicle.vin, interval="year")
pprint(yearly_stats)
loop = asyncio.get_event_loop()
loop.run_until_complete(get_information())
loop.close()
So it seems that we don't have to look for a new API endpoint for the trip statistics (for now).
This gist will get you connected to the new toyota services. You need to enter your username, password and GUID at the top of the file. The problem is the GUID this is your customer ID but I have yet to work out how the toyota app knows what this is! You can find it by logging into My Toyota website and in the developer tools find a request for https://cpb2cs.toyota-europe.com/vehicle/user/<This Will Be Your GUID>/...
. It may contain much more than is required but allows to me check the two accounts I have access too but is clearly a work in progress.
This gist will get you connected to the new toyota services. [...]
Hi @GitOldGrumpy, good finding! The system behind https://b2c-login.toyota-europe.com/
is probably from ForgeRock AS and is only used for identity and access management.
There is also API documentation for this here: https://backstage.forgerock.com/docs/am/7.4
I don't think that the telemetry data is also available via this endpoint.
By the way: Authentication is also still possible via https://ssoms.toyota-europe.com/
😉
Yep it is ForgeRock AS and once you have your access token it changes URL from https://b2c-login.toyota-europe.com
to https://ctpa-oneapi.tceu-ctp-prd.toyotaconnectedeurope.io
. This has many endpoints to pull information from but seems little point moving forward until I have worked out how to automatically get hold of the GUID(customerId). Can post up later what endpoints I've found and data it is responding with.
What is the GUID (customerId) format? I haven't been able to locate the https://cpb2cs.toyota-europe.com/vehicle/user/<This Will Be Your GUID>/...
request, but https://ssoms.toyota-europe.com
authentication provides both a uuid
and a myToyotaId
. The latter is a combination of three letters and five digits (abc12345
). I take it that the GUID
is neither of those?
I start at https://www.toyota.co.uk/customer/portal#/sso/login/autoclose=true
which logs you in and shows you your cars. I then click on my car(I have developer tools open) and it loads the details for the car. In the developer tools network view there are two requests for a URL starting with my cars Vin number if you look at the request URL under headers you should see the GUID. Its format is xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
.
Hopefully you can find it and then confirm that it is working for you.
I think it is using the UUID from the single sign on as GUID @GitOldGrumpy. Please have a look at the tojota repo from calmjm: https://github.com/calmjm/tojota/blob/ed49d83fd6cf2baf1bf528e40c58abd4a461b0c4/tojota.py#L157
I've just had a look the UUID is the GUID but the access token does not work, which is to be expected. Still confused on where the new app pulls this from...
@CM000n Sounds good if this fixes it! But I don't have an answer regarding ha_toyota
. Some part of it is ported over on this branch https://github.com/DurgNomis-drol/ha_toyota/tree/v2, but I never finished it. It needs a bit of work to get working. Again, it stalled because I don't have car to test most of the features on. If any one wants to pick it up, they are more then welcome to. And I will help get it working, but right now it is just to much of a task when I can't test most of it locally.
Updated the gist with some of the endpoints I've found but for some reason I'm not authenticated for them all.
Hi, I was migrated also with my Yaris Cross (2023) (works with new status API endpoint) and RAV4 Plugin (2022) (works only with legacy status endpoint) and HA integration stopped working as described here in comments. However when testing API using mytoyota python lib, I was able to get all needed information. Same with calling directly APIs. Unfortunately I didn't stored all API responses to compare but looks like smart info is missing (status info about locks, lights etc.) so maybe for that is now only special API endpoint.
API endpoint for "connected services": https://cpb2cs.toyota-europe.com/vehicle/user/{{uid}}/vehicle/{{VIN_RAV4}}?legacy=true&services=fud,connected this endpoint returns: "connected": false, "connectedService": {"status": "DISABLED"...}
so maybe this is the reason why HA integration states that connected services are offline? EDIT: I see @CM000n already fixed it with https://github.com/DurgNomis-drol/mytoyota/pull/240 👍
anyhow all other APIs worked well:
location: https://myt-agg.toyota-europe.com/cma/api/users/{{uid}}/vehicle/location ODO: https://myt-agg.toyota-europe.com/cma/api/vehicle/{{VIN_RAV4}}/addtionalInfo status: https://myt-agg.toyota-europe.com/cma/api/users/{{uid}}/vehicles/{{VIN_RAV4}}/vehicleStatus legacy status: https://myt-agg.toyota-europe.com/cma/api/vehicles/{{VIN_RAV4}}/remoteControl/status trips: https://cpb2cs.toyota-europe.com/api/user/{{uid}}/cms/trips/v2/history/vin/{{VIN_RAV4}}/1 trip: https://cpb2cs.toyota-europe.com/api/user/{{uid}}/cms/trips/v2/{{trip_ID}}/events/vin/{{VIN_RAV4}} driving stats: https://myt-agg.toyota-europe.com/cma/api/v2/trips/summarize?from=2023-01-01&calendarInterval=month
as stated cannot compare API results but one big difference I see is that "status endpoint" for new cars now doesn't show lock/light statuses... results is exactly same as for older vehicle...
I had one issue before with my RAV4 that it didn't show any legacy information which was available in mytoyota lib (e.g. it showed only EvTravelableDistance but not GasolineTravelableDistance and also all other info available here in json generated by mytoyota python lib was missing... Thats why I asked in another github issue to update HA integration to latest mytoyota plugin (with hope that more legacy data will be shown)... I don't have knowledge how to do it yet. Already tried to force HA use latest mytoyota python lib, but HA integration started to return errors and didn't work at all so there might be some compatibility issues when using latest mytoyota lib (0.9.2) and the one used by HA_toyota integration (0.7.8). So would be great to first get HA_toyota use latest 0.9.2 lib and then we can continue fixing migration...
"VehicleInfo": {
"AcquisitionDatetime": "2023-11-09T13:48:27Z",
"RemoteHvacInfo": {
"InsideTemperature": 25,
"RemoteHvacMode": 0,
"RemoteHvacProhibitionSignal": 1,
"SettingTemperature": 21.0,
"BlowerStatus": 0,
"FrontDefoggerStatus": 0,
"RearDefoggerStatus": 0,
"LatestAcStartTime": "2023-11-09T05:39:15Z",
"TemperatureDisplayFlag": 1,
"Temperaturelevel": 32
},
"ChargeInfo": {
"GasolineTravelableDistanceUnit": 1,
"GasolineTravelableDistance": 507,
"PlugInHistory": 33,
"RemainingChargeTime": 65535,
"EvTravelableDistance": 0.0,
"EvTravelableDistanceSubtractionRate": 2,
"ChargeRemainingAmount": 30,
"SettingChangeAcceptanceStatus": 0,
"ChargeType": 15,
"ChargeWeek": 0,
"ChargeStartTime": "42:35",
"ChargeEndTime": "42:35",
"ConnectorStatus": 2,
"BatteryPowerSupplyPossibleTime": 16383,
"ChargingStatus": "none",
"EvDistanceWithAirCoInKm": 0.0,
"PlugStatus": 12,
"EvDistanceInKm": 0.0
}
},
"ReturnCode": "000000"
Hi @DurgNomis-drol and @HuffYk,
I just created a tag for testing the update to mytoyota >= 0.9 within the ha_toyota custom integration: https://github.com/DurgNomis-drol/ha_toyota/releases/tag/v2.0.0-testing The tag uses the V2 branch as a basis. I think this makes testing easier in combination with HACS. I hope that's okay @DurgNomis-drol?
I also don't have much time at the moment, but I'll see how much of it I can spare for ha_toyota in the meantime :wink:
Unfortunately updating to the new mytoyota version it's a bit of trial and error because we don't have good unit tests for the custom integration and writing them is unfortunately a bit of a PITA. So any help is more than welcome!
Hi @CM000n, did few tests.
Looks like some changes from master branch have to be merged first. e.g. I had to fix deprecated async_setup_platforms with
await hass.config_entries.async_forward_entry_setups(entry, PLATFORMS)
Maybe some other changes from master should be merged also? I temporary fixed mytoyota lib to overcome "connected services" issue and now HA integration connected successfully, but it received only location for each car nothing else. And in log there are these errors (enabled also debugging):
ERROR (MainThread) [homeassistant.setup] Unable to prepare setup for platform toyota.binary_sensor: Platform not found (cannot import name 'ToyotaEntityDescription' from 'custom_components.toyota.entity' (/config/custom_components/toyota/entity.py)).
ERROR (MainThread) [homeassistant.components.sensor] Error while setting up toyota platform for sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 359, in _async_setup_platform
await asyncio.shield(task)
File "/config/custom_components/toyota/sensor.py", line 270, in async_setup_entry
ToyotaStatisticsSensor(
File "/config/custom_components/toyota/sensor.py", line 321, in __init__
self.period = description.period
^^^^^^^^^^^^^^^^^^
AttributeError: 'ToyotaSensorEntityDescription' object has no attribute 'period'
not sure if it is because something has to be merged from master what was already fixed or it is connected with new mytoyota lib or it is connected with new API json format :/
will continue tomorrow with more tests...
Hi @CM000n, did few tests.
Looks like some changes from master branch have to be merged first. e.g. I had to fix deprecated async_setup_platforms with
The master branch ist already merged 😉
Hi @CM000n
If I compare e.g. __init__.py
from master (https://github.com/DurgNomis-drol/ha_toyota/pull/131 ) with v2.0.0-testing there are many changes missing e.g. mentioned
await hass.config_entries.async_forward_entry_setups(entry, PLATFORMS)
introduced in v1.2.11 release (https://github.com/DurgNomis-drol/ha_toyota/pull/126/files )
did some testing with official v1.2.11, I patched mytoyota lib to overcome "connected services" issue and now I was able to receive other sensors:
RAV4 Plugin (2022) - older legacy API (not available status of locks or lights in API)
missing sensors what I remember:
Yaris Cross (2023) - new API (status of locks and lights is available in API)
missing sensors what I remember:
So for anyone who wants to have quick fix with current HA integration, here are steps to patch mytoyota lib in HASSIO running in Docker (debian):
sudo docker exec -it homeassistant bash
cd /usr/local/lib/python3.11/site-packages/mytoyota
vi vehicle.py
def _has_connected_services_enabled
function and replace this: _LOGGER.error(
"Please setup Connected Services if you want live data from the car. (%s)",
censor_vin(self.vin),
)
return False
with this:
return True
Hi @CM000n
If I compare e.g.
__init__.py
from master (#131 ) with v2.0.0-testing there are many changes missing e.g. mentioned [...]
Yes, I think some of the incoming and existing changes were lost during the merge when the merge conflicts were resolved. We need to look at this again in general and sort it out.
I've just had a look the UUID is the GUID but the access token does not work, which is to be expected. Still confused on where the new app pulls this from...
Hi @GitOldGrumpy
I use e.g. this: https://cpb2cs.toyota-europe.com/vehicle/user/{{uid}}/vehicles
and uid is coming from https://ssoms.toyota-europe.com/authenticate
as @CM000n wrote...
however token from ssoms doesn't work for https://ctpa-oneapi.tceu-ctp-prd.toyotaconnectedeurope.io
nor https://ctpa-oneapi.tceu-ctp-dev.toyotaconnectedeurope.dev
- locate
def _has_connected_services_enabled
function and replace this:_LOGGER.error( "Please setup Connected Services if you want live data from the car. (%s)", censor_vin(self.vin), ) return False
with this:
return True
That's great! With this "dirty" fix and v1.2.11, I get the data I need: I use the integration for controlling the charging (it's a bZ4x, so an EV), and I can find the "remaining charge" as a attribute to the EV battery status
sensor. This temporarily solves the problem for me, but I'm hoping we'll be able to fix the integration properly in the days to come 👍
Hi @CM000n
would it be possible to add back v2.0.0-testing tag ? I would like to test during weekend, but not in my production env (where I installed it yesterday), but rather I created dev env in RPI so would like to install it there from UI rather than manually updating all needed files in shell. thanks!
Hi @HuffYk,
No, the tags don't make much sense, as I found out when so many changes are made. The best way to test this is to pull the latest version of the Repo / V2 branch and manually replace the "toyota" folder in the custom components directory of your Home Assistant installation.
ok understand, no problem
@CM000n let me know if anything is ready for testing. I tried now latest V2 and getting this errors in log:
2023-11-12 00:28:57.005 ERROR (MainThread) [homeassistant.components.binary_sensor] Error while setting up toyota platform for binary_sensor
File "/config/custom_components/toyota/binary_sensor.py", line 307, in async_setup_entry
ToyotaBinarySensor(
File "/config/custom_components/toyota/entity.py", line 30, in __init__
self.vehicle: Vehicle = coordinator.data[self.index]["data"]
~~~~~~~~~~~~~~~~^^^^^^^^^^^^
TypeError: list indices must be integers or slices, not Vehicle
and few TypeError: 'NoneType' object is not callable
errors I suppose related to 1st one mentioned above.
In UI I'm able to see locations and few sensors all with unavailable or unknown status.
Hi @HuffYk, thank you very much. I am already aware of that. Please see the comment in the associated pull request. I think we should move further discussions regarding the custom integration update to the new mytoyota version there, as they are no longer directly related to the issue reported here.
Were have we moved discussions too? I can pull from the new endpoints all most all of the information relevant to my cars(which is everything I could get on the old endpoints plus all the trip details) but to move forward need some help getting hold of further endpoints and either integrating the new endpoint into the current mytoyota or creating something new?
Please open an PR in https://github.com/DurgNomis-drol/mytoyota so we can start implementing the new API endpoints
Describe the bug The Toyota integration has stopped working, seems to have happened at the same time as the car was moved (by Toyota) from the «MyT Toyota» app to the «MyToyota» app. I get this error message in the logs: