Mazda Connected Service Down

[danlo315]

The problem

Issue started today Aug-23. Getting the following log:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mazda/__init__.py", line 53, in with_timeout
    return await task
  File "/usr/local/lib/python3.10/site-packages/pymazda/client.py", line 27, in get_vehicles
    vec_base_infos_response = await self.controller.get_vec_base_infos()
  File "/usr/local/lib/python3.10/site-packages/pymazda/controller.py", line 21, in get_vec_base_infos
    return await self.connection.api_request("POST", "remoteServices/getVecBaseInfos/v4", body_dict={"internaluserid": "__INTERNAL_ID__"}, needs_keys=True, needs_auth=True)
  File "/usr/local/lib/python3.10/site-packages/pymazda/connection.py", line 162, in api_request
    return await self.__api_request_retry(method, uri, query_dict, body_dict, needs_keys, needs_auth, num_retries=0)
  File "/usr/local/lib/python3.10/site-packages/pymazda/connection.py", line 177, in __api_request_retry
    return await self.__send_api_request(method, uri, query_dict, body_dict, needs_keys, needs_auth)
  File "/usr/local/lib/python3.10/site-packages/pymazda/connection.py", line 231, in __send_api_request
    response = await self._session.request(method, self.base_url + uri, headers=headers, data=encrypted_body_Str, ssl=ssl_context)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File "/usr/local/lib/python3.10/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mazda/__init__.py", line 150, in async_update_data
    vehicles = await with_timeout(mazda_client.get_vehicles())
  File "/usr/src/homeassistant/homeassistant/components/mazda/__init__.py", line 52, in with_timeout
    async with async_timeout.timeout(timeout_seconds):
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

What version of Home Assistant Core has the issue?

2022.8.2

What was the last working version of Home Assistant Core?

2022.8.2

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Mazda Connected Service

Link to integration documentation on our website

https://www.home-assistant.io/integrations/mazda/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[probot-home-assistant[bot]]

mazda documentation mazda source _{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @bdr99, mind taking a look at this issue as it has been labeled with an integration (mazda) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

[hogantg]

Can confirm same error starting this morning 8/23 on HA core-2022.7.7

[Lennny]

I have the exact same issue this morning. 8/23 on HA core-2022.8.6. Seem to be the exact same thing.

here are my logs

Unknown error occurred during Mazda update request: Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/mazda/init.py", line 53, in with_timeout return await task File "/usr/local/lib/python3.10/site-packages/pymazda/client.py", line 27, in get_vehicles vec_base_infos_response = await self.controller.get_vec_base_infos() File "/usr/local/lib/python3.10/site-packages/pymazda/controller.py", line 21, in get_vec_base_infos return await self.connection.api_request("POST", "remoteServices/getVecBaseInfos/v4", body_dict={"internaluserid": "INTERNAL_ID__"}, needs_keys=True, needs_auth=True) File "/usr/local/lib/python3.10/site-packages/pymazda/connection.py", line 162, in api_request return await self.api_request_retry(method, uri, query_dict, body_dict, needs_keys, needs_auth, num_retries=0) File "/usr/local/lib/python3.10/site-packages/pymazda/connection.py", line 177, in api_request_retry return await self.send_api_request(method, uri, query_dict, body_dict, needs_keys, needs_auth) File "/usr/local/lib/python3.10/site-packages/pymazda/connection.py", line 231, in __send_api_request response = await self._session.request(method, self.base_url + uri, headers=headers, data=encrypted_body_Str, ssl=ssl_context) File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 559, in _request await resp.start(conn) File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 898, in start message, payload = await protocol.read() # type: ignore[union-attr] File "/usr/local/lib/python3.10/site-packages/aiohttp/streams.py", line 616, in read await self._waiter asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/mazda/init.py", line 150, in async_update_data vehicles = await with_timeout(mazda_client.get_vehicles()) File "/usr/src/homeassistant/homeassistant/components/mazda/init.py", line 52, in with_timeout async with async_timeout.timeout(timeout_seconds): File "/usr/local/lib/python3.10/site-packages/async_timeout/init.py", line 129, in aexit self._do_exit(exc_type) File "/usr/local/lib/python3.10/site-packages/async_timeout/init.py", line 212, in _do_exit raise asyncio.TimeoutError asyncio.exceptions.TimeoutError

[bdr99]

Thanks to everyone who has reported this. Recently, Mazda has been changing their API in an apparent attempt to block automated requests like the ones from this integration. So far, I have been able to find workarounds every time, but this time, it seems the changes they made are pretty drastic and I'm not sure if I will be able to work around it. It's possible this could be the end of the Mazda integration. 🙁 I'll still take a closer look, and will comment here when I have any updates.

If anyone has any experience with working around TLS fingerprinting in Python, please get in touch 🙂

[bdr99]

To anyone who was having this issue: Can you confirm whether the issue is still happening? I was seeing requests getting blocked earlier today, but now it looks like things are back to normal (you might need to restart HA). I think they might have reverted whatever change they made.

[danlo315]

I can confirm that the connected service is back up and running.

[Lennny]

Seems to be working ugh. this means they are messing with the api. Hopefully they dont block us out.

[hogantg]

An emotional rollercoaster this morning.

Can also confirm integration is up and running on HA Core-2022.7.7

[bdr99]

Thanks! It certainly has been an emotional rollercoaster. My theory is that the restrictions they added this morning were too strict and were blocking some legitimate users as well, so they had to revert them. It's not a good sign that they are experimenting with these types of restrictions.

[hogantg]

@bdr99 Can you share some details around the restrictions and TLS Fingerprinting you mentioned? I, admittedly, have very basic python knowledge, but love learning new things in this space.

[bdr99]

@hogantg TLS is the technology used to establish a secure connection with a web server. Whenever a URL begins with "https", TLS is being used. TLS fingerprinting is a technique used by APIs to try to distinguish between legitimate requests (from a browser, mobile app, etc) and automated/bot requests (from a programming language like Python). It works by analyzing patterns in the TLS handshake between the client and the server to try to identify which OS/browser/language was used to make the request.

Mazda recently started blocking requests which, according to the TLS fingerprint, were made by Python. I know this because I have captured a successful request made by the official MyMazda mobile app, and tried to make the same exact request from a Python script, but it failed. Also, the same request made from a Node.js program works fine. So, in order to work around this, we would need to change the Python code so that it performs the TLS handshake in such a way that it appears like Node.js or the MyMazda app. Unfortunately, Python doesn't seem to allow much control over the specifics of how it establishes the TLS connection. I am able to change the ciphers, which I do here, but not much else can be customized. Changing the ciphers that worked for some time, but this morning it seemed as though Mazda was able to block those requests as well. Since Python doesn't allow customizing the TLS handshake very much, there isn't much we can do if Mazda is able to recognize and block requests coming from Python.

[hogantg]

@bdr99 Is this a scenario where impersonating a browser comes into play? Some sort spoofing like https://github.com/lwthiker/curl-impersonate.git could do? Similarly, what stops python scripts from executing a node.js request? Just my curiosity - These things fascinate me as I am always eager to learn more, specifically about this integration. However, if you dont want to sit here and answer what may be dumb questions, I understand.

[bdr99]

@hogantg No problem, I don't mind at all. The more people who are thinking about this, the better chance we have at finding a solution!

That curl-impersonate library actually looks pretty helpful, but I'm not sure if it can be used in a Home Assistant context because it would require curl-impersonate to be installed on the host system where HA is running. Similarly, in order for Python to execute the requests through Node.js, it would require Node.js to be installed on the host system, which is not something we can guarantee, especially with Home Assistant OS which is the most common HA installation method.

[hogantg]

@bdr99 Those are very valid restrictions. I am new to most of HA and totally forgot about other installation methods as I am running HA Core in a docker container. Question regarding integration installations... I would assume that many integrations use more than just the pre-installed python libraries. When an integration is installed by HA is there not a pre-requisite check the system does to install required libraries? The way your responses is worded makes me think not. In that train of thought, my assumption that integrations use third-party libraries may very well be wrong.

In a world where the API locks you out due to these TLS fingerprinting issues, would a HACS type install with manual-user installation of Node or curl-impersonate be an avenue to keep the integration active? And, would this limit the integration to non-HA OS installations only? My limited research just now shows a seeming lack of Node setup documentation on HA OS.

Interestingly, the Mazda integration showed unavailable 13 times this morning, 12 instances less than 5 minutes and one for 9 minutes. It is currently 'live', but wondering if you see the same thing in your HA history tab querying MazdaGPS.

Edit: Just a random update for anyone that comes across this post. Over the 48 hours following the morning mentioned above the integration went offline ~65 times. Only a handful for longer than 5 minutes. Interesting to say the least. Curious if they were testing API changes or not. Since that 48 hours the integration has been live with no interruptions!

[bdr99]

@hogantg Sorry for the delay in replying. Integrations do use third party libraries - every HA integration has a manifest.json file which includes, among other things, the list of dependencies for that integration. You can check them out here - see the manifest.json in each folder. When the user adds an integration, HA retrieves the dependencies from PyPI which is Python's package repository. However, only Python libraries can be installed through this method. There is no way for HA to automatically install system libraries that run outside of a Python environment, so that's why I don't think curl-impersonate would work.

Moving the integration to HACS and requiring the user to manually install any necessary libraries like curl-impersonate might be an option. I could also create a way to proxy the requests from the HA integration through a Node.js program, since Node.js requests are working, but that would also require moving the integration to HACS since I don't think the HA team would want a built-in integration to require a separate program to be running. Another potential option would be to create an add-on which would allow installing custom software that is separate from the Python environment where HA is running. However, I think this could be a bit complex to set up, and it would also not support the HA Container or Core installation methods since Add-ons only work with Supervised or HA OS.

[hogantg]

In case anyone comes across this now - integration has been unavailable with the same error since 4:30 PM CST 9/8/22.

[bdr99]

Thanks @hogantg for reporting that. Hopefully it will be another temporary change.

[hogantg]

Back up! 26.5 hours down. Now up since 7pm CST 9/9!

[StephenMilner]

Having similar issues today, intermittently - from 11am UTC

Although the app is reporting an [E900500] error intermittently as well.

AWS seem to be having outages so if the Connect platform is hosted on AWS that could be the reason. (Although I can't work out if I'm looking at current reporting so this could be a red herring)

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.