Problems with login from Home Assistant - http 403

[mattsch]

I deleted the integration when errors started occurring, restarted Hass, and then tried to re-add. Logs attached from my attempt to re-add. FWIW, had the same problem with and without MFA enabled.

Reported here as well.

home-assistant.log

[alandtse]

Thanks. This is already fixed and waiting for merging.

[alandtse]

Reopening as there is a 403 block we need to work around. It affects the oauth login too.

[mattsch]

Yup, I've been trying to figure out why and am coming up completely blank. I can use curl to grab the same URL with the same exact headers (including user agent) and it works perfectly. Once I try from within the existing HA integration (so no proxy), that 403 comes up. Every. Time. I can't for the life of me figure out why the response is different.

[alandtse]

Yah welcome to my world. 😉 Right now the same code running on my dev server running on my mac book works but the same code on my rpi gets the 403.

[mattsch]

It's really frustrating. I even tried running curl from inside my HA container and it passed. It's impressively painful to debug.

[alandtse]

I'm concerned there may be something up with aiohttp. It's not the first time I've seen weird behavior there.

[mattsch]

Was thinking the same thing. Using the requests library does work. A simple r.get() to the same URL that fails using aiohttp works just fine without setting extra headers or anything of the sort. Mind-boggling to think that it might actually come down to some sort of weird underlying library difference.

[alandtse]

Be aware that aiohttp does autogenerate headers so that could be coming in play. The proxy code allows you to turn it off. https://auth-capture-proxy.readthedocs.io/en/latest/autoapi/authcaptureproxy/auth_capture_proxy/index.html#authcaptureproxy.auth_capture_proxy.AuthCaptureProxy.modify_headers

[mattsch]

I was manually setting the headers, and resp.request_info.headers was showing the various values I was trying.

[mattsch]

From a data and header perspective, I can't find a difference between aiohttp and requests/curl. I spun up a simple webserver to point all three at so I could dump the traffic more easily and it was exactly the same. My best guess at this point is it has to do with a lower level implementation, perhaps aiohttp is a bit more aggressive about how it connects which triggers the WAF's scanning protection? Or perhaps it's something entirely different....

[alandtse]

Thanks for investigating that. This is perplexing. I'm debating whether we include requests just for the auth but it seems to be a step backwards to include two libraries.

[alandtse]

And my dev server still works in case you want to see the response when it works.

2021-04-23 04:43:16 DEBUG (MainThread) [authcaptureproxy.auth_capture_proxy] Received get: http://TESTSERVER/auth/tesla/proxy?config_flow_id=f74f6c541a03467b8c5679b5bd56202d&callback_url=https://TESTSERVER/auth/tesla/callback?flow_id%3Df74f6c541a03467b8c5679b5bd56202d for https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=YjA1ZDhkZGQ2ZDk4ZWMwYWFjYzMwZWE1Y2RkYjQyYWE5NjAzNDMxYzJmYzQyOGI4OGViODg2NjFhMmFkMTdjNw%3D%3D&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=SNIPnXBEqFL2gsOpR-6Lyv1VE6ImFVpob3ui10nQ
2021-04-23 04:43:16 DEBUG (MainThread) [authcaptureproxy.auth_capture_proxy] Detected http while should be https; switching to https
2021-04-23 04:43:16 DEBUG (MainThread) [authcaptureproxy.auth_capture_proxy] Starting auth capture proxy for https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=YjA1ZDhkZGQ2ZDk4ZWMwYWFjYzMwZWE1Y2RkYjQyYWE5NjAzNDMxYzJmYzQyOGI4OGViODg2NjFhMmFkMTdjNw%3D%3D&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=SNIPnXBEqFL2gsOpR-6Lyv1VE6ImFVpob3ui10nQ
2021-04-23 04:43:16 DEBUG (MainThread) [authcaptureproxy.auth_capture_proxy] Discovered skip_auto_headers ['User-Agent']
2021-04-23 04:43:16 DEBUG (MainThread) [authcaptureproxy.auth_capture_proxy] Attempting get to https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=YjA1ZDhkZGQ2ZDk4ZWMwYWFjYzMwZWE1Y2RkYjQyYWE5NjAzNDMxYzJmYzQyOGI4OGViODg2NjFhMmFkMTdjNw%3D%3D&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=SNIPnXBEqFL2gsOpR-6Lyv1VE6ImFVpob3ui10nQ
headers: <MultiDict('sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"', 'sec-ch-ua-mobile': '?0', 'upgrade-insecure-requests': '1', 'dnt': '1', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'sec-fetch-site': 'cross-site', 'sec-fetch-mode': 'navigate', 'sec-fetch-user': '?1', 'sec-fetch-dest': 'document', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,ja;q=0.6,mt;q=0.5', 'Cookie': 'i18next=en-US; csm-hit=tb:DZ58MMHSBJNYFNG90NSM+s-GK3CTCNQJSJER70WP2YW|1618088588686&t:1618088588686&adb:adblk_yes', 'x-tesla-user-agent': 'TeslaApp/3.10.9-433/adff2e065/android/10', 'X-Requested-With': 'com.teslamotors.tesla')>
cookies:
2021-04-23 04:43:16 DEBUG (MainThread) [authcaptureproxy.helper] GET:
https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=YjA1ZDhkZGQ2ZDk4ZWMwYWFjYzMwZWE1Y2RkYjQyYWE5NjAzNDMxYzJmYzQyOGI4OGViODg2NjFhMmFkMTdjNw%3D%3D&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=SNIPnXBEqFL2gsOpR-6Lyv1VE6ImFVpob3ui10nQ with
{"Host": "auth.tesla.com", "sec-ch-ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\"", "sec-ch-ua-mobile": "?0", "upgrade-insecure-requests": "1", "dnt": "1", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9", "sec-fetch-site": "cross-site", "sec-fetch-mode": "navigate", "sec-fetch-user": "?1", "sec-fetch-dest": "document", "Accept-Encoding": "gzip, deflate, br", "Accept-Language": "en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,ja;q=0.6,mt;q=0.5", "Cookie": {"i18next": "en-US", "csm-hit": "tb:DZ58MMHSBJNYFNG90NSM+s-GK3CTCNQJSJER70WP2YW|1618088588686&t:1618088588686&adb:adblk_yes"}, "x-tesla-user-agent": "TeslaApp/3.10.9-433/adff2e065/android/10", "X-Requested-With": "com.teslamotors.tesla"}
returned 200:OK with response <CIMultiDictProxy('Server': 'nginx', 'Content-Type': 'text/html; charset=utf-8', 'X-DNS-Prefetch-Control': 'off', 'X-Frame-Options': 'DENY', 'Strict-Transport-Security': 'max-age=15552000; includeSubDomains', 'X-Download-Options': 'noopen', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'X-Request-ID': '7ce559af-1723-4a7c-a710-015052702ad3', 'X-Correlation-ID': '7ce559af-1723-4a7c-a710-015052702ad3', 'Content-Security-Policy': "connect-src 'self'; default-src 'none'; font-src 'self' data: fonts.gstatic.com; frame-src 'self' www.google.com www.recaptcha.net; img-src 'self' data:; script-src www.recaptcha.net 'self' 'nonce-a4068e9d966df33becc1'; style-src 'unsafe-inline' 'self'", 'Etag': 'W/"663b-Ibgivb3j1Q4K8N9T9M64WJWUIQA"', 'X-Response-Time': '20.759ms', 'X-EdgeConnect-MidMile-RTT': '50', 'X-EdgeConnect-Origin-MEX-Latency': '75', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Date': 'Fri, 23 Apr 2021 04:43:16 GMT', 'Content-Length': '5337', 'Connection': 'keep-alive', 'Set-Cookie': 'tesla-auth.sid=s%3AhxScs_GQUH4HmFRuAwsR3BSW_X4WduFH.o4l2CfPpyljYb%2FpBQJZFyB1N29aJnOt8j6JRXtk9wLg; Path=/; Expires=Mon, 26 Apr 2021 04:43:16 GMT; HttpOnly; Secure; SameSite=Lax')>

root ➜ /workspaces $ pip list | egrep "teslajsonpy|aiohttp|authcaptureproxy"
aiohttp                     3.7.4.post0
aiohttp-cors                0.7.0
authcaptureproxy            0.8.1
pytest-aiohttp              0.3.0
teslajsonpy                 0.17.1

The only difference right now is it's directly overwritten the tesla component instead of using custom_components. I guess I can try that route too.

EDIT: Doesn't matter if it's a custom component. I used pr_custom_component on my dev server and it still works there too.

For good measure, here's my failing production server:

2021-04-22 22:35:41 DEBUG (MainThread) [authcaptureproxy.auth_capture_proxy] Received get: http://PROD/auth/tesla/proxy?config_flow_id=e09c6e8d8339449296da5b4bddad9914&callback_url=https://PROD/auth/tesla/callback?flow_id%3De09c6e8d8339449296da5b4bddad9914 for https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NTI3ZjY0MGIxOGIxYWExNjY5Yzc4M2E4MTlkNTgwZDRhYTkxOTJlNmFkMDQzZjYyNzg1Njk0NzdkNTcxODc5YQ%3D%3D&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=SNIPMwIQCQ7QgFUGMAhD9xun4D6S1NBmbzQUbT5Vc0mw07DHz8jH-xQ
2021-04-22 22:35:41 DEBUG (MainThread) [authcaptureproxy.auth_capture_proxy] Detected http while should be https; switching to https
2021-04-22 22:35:41 DEBUG (MainThread) [authcaptureproxy.auth_capture_proxy] Starting auth capture proxy for https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NTI3ZjY0MGIxOGIxYWExNjY5Yzc4M2E4MTlkNTgwZDRhYTkxOTJlNmFkMDQzZjYyNzg1Njk0NzdkNTcxODc5YQ%3D%3D&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=SNIPMwIQCQ7QgFUGMAhD9xun4D6S1NBmbzQUbT5Vc0mw07DHz8jH-xQ
2021-04-22 22:35:41 DEBUG (MainThread) [authcaptureproxy.auth_capture_proxy] Discovered skip_auto_headers ['User-Agent']
2021-04-22 22:35:41 DEBUG (MainThread) [authcaptureproxy.auth_capture_proxy] Attempting get to https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NTI3ZjY0MGIxOGIxYWExNjY5Yzc4M2E4MTlkNTgwZDRhYTkxOTJlNmFkMDQzZjYyNzg1Njk0NzdkNTcxODc5YQ%3D%3D&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=SNIPMwIQCQ7QgFUGMAhD9xun4D6S1NBmbzQUbT5Vc0mw07DHz8jH-xQ
headers: <MultiDict('X-Real-IP': '192.168.1.1', 'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'dnt': '1', 'upgrade-insecure-requests': '1', 'sec-ch-ua-mobile': '?0', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36', 'sec-fetch-site': 'same-origin', 'sec-fetch-mode': 'same-origin', 'sec-fetch-dest': 'empty', 'Referer': 'https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NTI3ZjY0MGIxOGIxYWExNjY5Yzc4M2E4MTlkNTgwZDRhYTkxOTJlNmFkMDQzZjYyNzg1Njk0NzdkNTcxODc5YQ%3D%3D&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=SNIPMwIQCQ7QgFUGMAhD9xun4D6S1NBmbzQUbT5Vc0mw07DHz8jH-xQ', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,ja;q=0.6,mt;q=0.5', 'x-tesla-user-agent': 'TeslaApp/3.10.9-433/adff2e065/android/10', 'X-Requested-With': 'com.teslamotors.tesla')>
cookies:
2021-04-22 22:35:41 DEBUG (MainThread) [authcaptureproxy.helper] GET:
https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NTI3ZjY0MGIxOGIxYWExNjY5Yzc4M2E4MTlkNTgwZDRhYTkxOTJlNmFkMDQzZjYyNzg1Njk0NzdkNTcxODc5YQ%3D%3D&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=SNIPMwIQCQ7QgFUGMAhD9xun4D6S1NBmbzQUbT5Vc0mw07DHz8jH-xQ with
{"Host": "auth.tesla.com", "X-Real-IP": "192.168.1.1", "sec-ch-ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\"", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9", "dnt": "1", "upgrade-insecure-requests": "1", "sec-ch-ua-mobile": "?0", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36", "sec-fetch-site": "same-origin", "sec-fetch-mode": "same-origin", "sec-fetch-dest": "empty", "Referer": "https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NTI3ZjY0MGIxOGIxYWExNjY5Yzc4M2E4MTlkNTgwZDRhYTkxOTJlNmFkMDQzZjYyNzg1Njk0NzdkNTcxODc5YQ%3D%3D&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=SNIPMwIQCQ7QgFUGMAhD9xun4D6S1NBmbzQUbT5Vc0mw07DHz8jH-xQ", "Accept-Encoding": "gzip, deflate, br", "Accept-Language": "en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,ja;q=0.6,mt;q=0.5", "x-tesla-user-agent": "TeslaApp/3.10.9-433/adff2e065/android/10", "X-Requested-With": "com.teslamotors.tesla"}
returned 403:Forbidden with response <CIMultiDictProxy('Server': 'AkamaiGHost', 'Mime-Version': '1.0', 'Content-Type': 'text/html', 'Content-Length': '296', 'Expires': 'Fri, 23 Apr 2021 05:35:41 GMT', 'Date': 'Fri, 23 Apr 2021 05:35:41 GMT', 'Connection': 'close')>

bash-5.0# pip list | egrep "teslajsonpy|aiohttp|authcaptureproxy"

aiohttp                          3.7.4.post0
aiohttp-cors                     0.7.0
authcaptureproxy                 0.8.1
teslajsonpy                      0.17.1

[mattsch]

Is there a difference in OS and/or hardware specs between your dev and production servers? I'm struggling to explain why the behavior is different.

[alandtse]

Yes there's a difference but I'm struggling to understand why it would matter.

The dev server is a homeassistant docker container inside my osX M1 host.

Linux dcd78d32b46d 5.10.25-linuxkit #1 SMP PREEMPT Tue Mar 23 09:24:45 UTC 2021 x86_64 GNU/Linux

The production is a raspberry pi4 running the latest version of raspbian.

Linux rpi4 5.10.17-v7l+ #1403 SMP Mon Feb 22 11:33:35 GMT 2021 armv7l GNU/Linux

Please note the production server had previously logged in using the PR code and has a valid tokens session (for now). The new 403 is happening when I try to add the integration again (which is supported for multiple accounts).

[mattsch]

I was just wondering if it could have to do with performance somewhere in the stack. Faster machine, faster requests, WAF unhappy. But that M1 host should be snappier than the rpi4 for such things so that theory is right out.

[yurgh]

I wish I could contribute, but my python skills are at the hello world stage.

Could you try posting the authentication request to your own server and diff the requests from production and developement to see if there are indeed changes being made by aiohttp (or somewhere else).

The headers being sent in your two examples above are quite different. Are the sec-fetch headers neccessary for the API?

[alandtse]

I wish I could contribute, but my python skills are at the hello world stage.

Could you try posting the authentication request to your own server and diff the requests from production and developement to see if there are indeed changes being made by aiohttp (or somewhere else).

The headers being sent in your two examples above are quite different. Are the sec-fetch headers neccessary for the API?

@mattsch actually did that with his browser, curl and aiohttp and didn't see any differences. It's a bit of work to setup a fake Tesla server to intercept the traffic.

Are the sec-fetch headers neccessary for the API?

Who knows. I can remove them and test but mattsch also played with the headers too. They're coming from the browser but I'm literally using the same browser with different tabs to test.

[alandtse]

I've removed the headers and it doesn't matter. Also downgraded to aiohttp 3.7.3 in case something crazy happened there. Still no luck.

I just realized the error page is saying:

You don't have permission to access "http://auth.tesla.com/oauth2/v3/authorize?" on this server.

My logs indicate we're doing https in the get request. @mattsch Are you seeing the same weird message about http?

I also tried this toy example.

import aiohttp
import asyncio
async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('http://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NjRiN5ZmJkNTIyN2UwNzE4ZGExZTZkNmEwOGMyOTE4NzIyN2QyMjhiOWNkZWY2ZjI0YTRkZjhlNTVhMzQxOQ%3D%3D&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=EtpSQ8Wt4uw1fN5_mMF_t2I7T1JVGMW4T8gwzulHGATZuoZXbhdLZkAX53-pvYwN9X4siVIoNNWzD0BDbmMVQ') as resp:
            print(resp.status)
            print(await resp.text())

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

It gets the 403 page on my production server but not my dev server. On the host for my dev server, I get the 403.

[alandtse]

This toy example with requests is even wierder.

import aiohttp
import asyncio
import requests

URL = "http://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NjRiN5ZmJkNTIyN2UwNzE4ZGExZTZkNmEwOGMyOTE4NzIyN2QyMjhiOWNkZWY2ZjI0YTRkZjhlNTVhMzQxOQ%3D%3D&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=EtpSQ8Wt4uw1fN5_mMF_t2I7T1JVGMW4T8gwzulHGATZuoZXbhdLZkAX53-pvYwN9X4siVIoNNWzD0BDbmMVQ"

async def main(requests_header):
    async with aiohttp.ClientSession() as session:
        async with session.get(URL, headers=requests_header) as resp:
            print(resp.status)
            print(resp.request_info.headers)
            # print(await resp.text())

req = requests.get(URL)
requests_header = req.request.headers
print(req.status_code)
print(requests_header)

loop = asyncio.get_event_loop()
loop.run_until_complete(main(requests_header))

EDIT: I run a hassio docker on my production rpi4 server. This code will 403 in the docker container. On the baremetal rpi4. No 403. My dev machine is MacOS with a homeassistant docker. No 403 in docker. MacOS baremetal has a 403.

[yurgh]

I'm using Ubuntu 20.04.2 LTS bare metal box Linux ha 5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

jvb@ha:~/test$ pip3 list | egrep "teslajsonpy|aiohttp|authcaptureproxy"
aiohttp                3.7.4.post0
authcaptureproxy       0.8.1
teslajsonpy            0.17.1

Running that last script I get:

jvb@ha:~/test$ python3 test.py
200
{'User-Agent': 'python-requests/2.22.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
200
<CIMultiDictProxy('Host': 'auth.tesla.com', 'User-Agent': 'python-requests/2.22.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive')>

HTTP 200 is OK. Doesn't get less strange :)

[alandtse]

If the toy script works, the PR should also. Does it?

[mattsch]

@alandtse I did notice that the response said http instead of https as well in the 403 message, but I've just assumed it was a typo on their end. If I do point at that URL it just redirects me to the https version before kicking me out so I think it's a red herring.

Here's my slightly tweaked copy of the toy script. I am using requests.Session since I wanted to play with cookies a bit just to see if there was some odd interaction there, but that was a dead end. This aiohttp portion of the script works on a Pi 4, my Hass host (NUC running Fedora 33), but not in the docker container on that same host nor on my M1 MacBook Air. The requests portion works perfectly on all three. Oh, and that docker container is using host networking.

As you can see, the user-agent is the same in both, and I set the connection header to match what requests sets but I get the same behavior either way.

The only other variable I see here is python versions. My aiohttp versions are all the same so that's probably the next best place to start poking. I'll have some time a bit later dig, but for now here's what I'm running: Hass host: 3.9.2 (works) Hass container: 3.8.7 (fails) MacBook: 3.9.4 (fails) Pi 4: Python 3.7.3 (works)

#!/usr/bin/env python

import aiohttp
import asyncio
import requests

head = {
        "User-Agent": "python-requests/2.25.1",
        "Connection": "keep-alive"
        }
url = 'https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NDgwM2RlMGE1NTEwZWI1NWIzM2Q2NzM3YTRkYTBlZWNjYWMyOGUzZGZiNDJkNmZkNWE3ZDkxNmQ1MzI5YTg0OQ&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=9_MVz16nNle7FSB8-O50bKZId0XNAgTrrIg9agarIiPBV9GnMtsw3uAHeC3jXNjLjs4CSYrqQ5EQBIy-_fmoVQ'

async def fetch(client):
    async with client.get(
            url,
            headers = head) as resp:
        return resp

async def main():
    async with aiohttp.ClientSession() as client:
        r = await fetch(client)
        print("AIOHTTP")
        print("Request headers:", r.request_info.headers)
        print("Response Headers:", r.headers)
        print("Response code:", r.status)
        print("Full response:", r)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

sess = requests.Session()
req = sess.get(url)
print("REQUESTS")
print("Request headers:", req.request.headers)
print("Response headers:", req.headers)
print("Response code:", req.status_code)

# vim:syntax=python
# vim:sw=4:softtabstop=4:expandtab

[alandtse]

Here's a fun one. On my Macbook host (python 3.9.4) which fails normally, if I enable a mitmproxy it works. It's definitely aiohttp being rejected.

You'll have to set the proxy value to False below to disable the proxy. I set the proxy in my network connection and realized that aiohttp was going around the network setting. I had to use the proxy keyword to force it to use my proxy.

#!/usr/bin/env python

import aiohttp
import asyncio
import ssl
import requests

custom_context = ssl.SSLContext()

head = {"User-Agent": "python-requests/2.25.1", "Connection": "keep-alive"}
url = "https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NDgwM2RlMGE1NTEwZWI1NWIzM2Q2NzM3YTRkYTBlZWNjYWMyOGUzZGZiNDJkNmZkNWE3ZDkxNmQ1MzI5YTg0OQ&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=9_MVz16nNle7FSB8-O50bKZId0XNAgTrrIg9agarIiPBV9GnMtsw3uAHeC3jXNjLjs4CSYrqQ5EQBIy-_fmoVQ"

async def fetch(client, proxy=False):
    if proxy:
        async with client.get(
            url, headers=head, ssl=custom_context, proxy="http://localhost:8080"
        ) as resp:
            return resp
    async with client.get(url, headers=head) as resp:
        return resp

async def main():
    async with aiohttp.ClientSession() as client:
        r = await fetch(client, proxy=True)
        print("AIOHTTP")
        print("Request headers:", r.request_info.headers)
        print("Response Headers:", r.headers)
        print("Response code:", r.status)
        print("Full response:", r)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

sess = requests.Session()
req = sess.get(url, verify=False)
print("REQUESTS")
print("Request headers:", req.request.headers)
print("Response headers:", req.headers)
print("Response code:", req.status_code)

# vim:syntax=python
# vim:sw=4:softtabstop=4:expandtab

[yurgh]

As for Python versions, I tested 3.8.5, 3.8.7 and 3.9.4 (using pyenv) on Ubuntu 20 and they all worked with aiohttp and requests. The 403/forbidden is served by AkamaiGHost, so maybe the request is blocked before reaching Tesla? Maybe someone else also has this problem?

[mattsch]

@alandtse Good find. Can confirm the same behavior across my test hosts. With mitmproxy it works just fine, without it 403s. Now my question is why in the !*@&# does it not fail consistently?? If it were a blanket block somehow for the aiohttp client, it would fail everywhere and not just some places. I might just have to dig up docs on the WAF to see if there's any clues.

[alandtse]

I'm going to open an issue with aiohttp. I don't think mitmproxy uses aiohttp under the hood but I think we've demonstrated it's isolated to aiohttp.

The question of course it whether they can reproduce the 403. Is docker the common issue here? My case has the issue flip between host and guest so that's weird.

Edit: i did see ALPN is set differently with requests. Maybe that is related?

[alandtse]

As for Python versions, I tested 3.8.5, 3.8.7 and 3.9.4 (using pyenv) on Ubuntu 20 and they all worked with aiohttp and requests. The 403/forbidden is served by AkamaiGHost, so maybe the request is blocked before reaching Tesla? Maybe someone else also has this problem?

Are you able to test the actual Tesla PR? If the toy script works, the actual PR should be working for you. If you're saying it doesn't that's another weird case to consider.

[mattsch]

I'm going to open an issue with aiohttp. I don't think mitmproxy uses aiohttp under the hood but I think we've demonstrated it's isolated to aiohttp.

The question of course it whether they can reproduce the 403. Is docker the common issue here? My case has the issue flip between host and guest so that's weird.

Docker doesn't seem to matter in my testing. It fails on hosts and in containers. :(

Edit: i did see aapn is set differently with requests. Maybe that is related?

Edit 2 I may have the wrong term but check your mitmproxy logs. I'm not near my machine right now but will update when I'm back.

I don't think it matters since it doesn't pass through the ALPN connection request to the server:

What I do find interesting is the timings. The aiohttp connections are all sub 10ms, while request connections are all 40-70ms on my laptop. I can't really explain it completely just from that since presumably connections should be faster from my Hass host than from inside the container on that host, yet the failures are inverted.

Oh, and I removed the user-agent header completely so I could differentiate between the two in mitmproxy and it works with the auto-generated Python/3.9 aiohttp/3.7.4.post0 header. So that's definitely ruled out here.

[alandtse]

I don't think it matters since it doesn't pass through the ALPN connection request to the server:

Normally I'd agree. But that's like the only difference I can see between the aiohttp get vs the requests get. It's also something a WAF could see so maybe that's the trigger. I can't seem to find any documentation on setting a ALPN on aiohttp.

[mattsch]

If it was the culprit I'd expect that would be consistent, but it's not. And the server connection from mitmproxy also shows it's not setting the header and it succeeds.

[yurgh]

Are you able to test the actual Tesla PR? If the toy script works, the actual PR should be working for you. If you're saying it doesn't that's another weird case to consider.

Unfortunately I don't know how to merge the pull request from github into my docker container :-\ Edit: after pasting the changed files from my PC thru vi in the docker container it fails at the proxy part: Address: myexternalservername/auth/tesla/proxy?config_flow_id=895918b820b346e197ebcae03abc6ef2&callback_url=https://myexternalservername/auth/tesla/callback?flow_id%3D895918b820b346e197ebcae03abc6ef2 "Access Denied You don't have permission to access "http://auth.tesla.com/oauth2/v3/authorize?" on this server. Reference #18.5ede4568.1619297164.d7bb6f4

I guess I missed something with the hot-patching :)

[alandtse]

I guess I missed something with the hot-patching :)

And you said the toy script worked inside your docker? Can you please confirm because that is an additional weird case.

[yurgh]

And you said the toy script worked inside your docker? Can you please confirm because that is an additional weird case.

That's a negative - the toy script (https://github.com/zabuldon/teslajsonpy/issues/190#issuecomment-826101940) does not work in the container: aiohttp get a 403 forbidden response. Requests get 200 ok.

The container is set up with host networking, and on the host the toy script get a 200 ok response with both methods.

[alandtse]

Well that got closed quick. They said it they'll assume it's not an aiohttp bug and asked if we did any netcat. I don't know how to use netcat. Do you?

I may just rip out aiohttp and use another library. Since I'm not going to support HA's core integration anymore, I don't have to stay with aiohttp on the component either.

[mattsch]

I'm not sure netcat would gain us much here really. Tcpdump is probably the next best option to show the actual network layer differences. 😞

[alandtse]

Btw thanks for your help in tracking this down. It helps that someone else technical can reproduce it so I'm not just crazy. ;)

If we're really at the tcpdump level, I think your hypothesis about response time may be right. I did some more searching on the Akamai GHost and 403s and you will see this issue pop up for random items. I'm wondering if Akamai has created some logic such as DDOS protection that is getting tripped up by aiohttp. It's also possible that aiohttp has been used in such attacks in the past so it's already a close trigger.

[yurgh]

I came over some similar cases with DDOS protection triggering because the ordering of the request headers were off. If the host header wasn't first it would not get through. I will look into it in more depth tomorrow.

[hunterjm]

I may just rip out aiohttp and use another library. Since I'm not going to support HA's core integration anymore, I don't have to stay with aiohttp on the component either.

If you want to stick with an async library with a requests-esque API, may I suggest httpx. Home Assistant core already pulls that in as well because aiohttp doesn't support digest auth.

[alandtse]

I may just rip out aiohttp and use another library. Since I'm not going to support HA's core integration anymore, I don't have to stay with aiohttp on the component either.

If you want to stick with an async library with a requests-esque API, may I suggest httpx. Home Assistant core already pulls that in as well because aiohttp doesn't support digest auth.

Thanks. That was my planned replacement route. It's less performant than aiohttp but given the use case it's probably fine.

[mattsch]

httpx works perfectly across my environments. My vote is to ditch aiohttp if it fixes things for others as well.

#!/usr/bin/env python
# $Id$

import aiohttp
import asyncio
import requests
import ssl
import httpx

head = {
        "User-Agent": "python-requests/2.25.1",
        "Connection": "keep-alive"
        }
url = 'https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NDgwM2RlMGE1NTEwZWI1NWIzM2Q2NzM3YTRkYTBlZWNjYWMyOGUzZGZiNDJkNmZkNWE3ZDkxNmQ1MzI5YTg0OQ&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=9_MVz16nNle7FSB8-O50bKZId0XNAgTrrIg9agarIiPBV9GnMtsw3uAHeC3jXNjLjs4CSYrqQ5EQBIy-_fmoVQ'

nossl_conn = aiohttp.TCPConnector(ssl=False)
proxy = "http://localhost:8080"

async def fetch(client):
    async with client.get(
            url,
            #proxy=proxy
            ) as resp:
        return resp

async def main():
    async with aiohttp.ClientSession() as client:
        r = await fetch(client)
        print("AIOHTTP")
        print("Request headers:", r.request_info.headers)
        print("Response Headers:", r.headers)
        print("Response code:", r.status)
        print("Cookies:", r.cookies)

    async with httpx.AsyncClient() as clientx:
        rx = await clientx.get(url)
        print("HTTPX")
        print("Request headers:", rx.request.headers)
        print("Response Headers:", rx.headers)
        print("Response code:", rx.status_code)
        print("Cookies:", rx.cookies)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

#sess = requests.Session()
##sess.proxies = {"http": proxy, "https": proxy}
#sess.verify = False
#req = sess.get(url)
#print("REQUESTS")
#print("Request headers:", req.request.headers)
#print("Response headers:", req.headers)
#print("Response code:", req.status_code)

# vim:syntax=python
# vim:sw=4:softtabstop=4:expandtab

Edit: According to mitmproxy, the request timings of httpx are slightly faster (~10-15ms) than the requests library.

[yurgh]

Tested

HTTPX works both on bare-metal and in the docker container (where aiohttp fails)

[alandtse]

Swap completed to httpx and issues are gone with my testing. With any major rework, I may have missed something so I need testers. You will need the last rejected PR to HA. If you're on HA dev, it will work without edits. If you're not on dev, you'll need to back out some changes. See this post. I will eventually release my tesla using the proxy as a custom component but this api work took precedence.

You will manually have to take the PRs for authcaptureproxy and teslajsonpy.

I won't provide any support on installing things the above. A easier installation will probably become available later this week.

[mattsch]

@alandtse Can confirm it works!

[mattsch]

One problem I just ran into is that it fails to setup properly if there's an error when starting up. Doesn't look like httpx has any native support for retries so we might need some custom logic here to retry. And setting the connection timeout doesn't seem to actually help either for some reason. fail.log

Edit: Looks like you have to pass the timeout to the getattr calls at lines 200 and 203. Setting that to 60.0 works for me at startup. It's not seeing the seat warmers but that's another thing entirely. :)

[alandtse]

Edit: Looks like you have to pass the timeout to the getattr calls at lines 200 and 203. Setting that to 60.0 works for me at startup. It's not seeing the seat warmers but that's another thing entirely. :)

So you're saying the default of 5s doesn't work for you consistently? Or it never works? 60 seems like a big jump up though. I guess aiohttp was at 300 but that seems way too much.

[mattsch]

Yup, 5s doesn't work at all with one car online and the other asleep. It's not a problem with both asleep since it doesn't go grabbing all the data, but seems to be a bit slower doing that call than the initial vehicles list. Without some retry logic, we may want to go even longer than 60 seconds just to be on the safe side.

[alandtse]

This is resolved in https://github.com/alandtse/home-assistant/tree/tesla_oauth_callback. I will be packaging it as a custom component later this week.

[yurgh]

Thanks for your efforts in keeping my car connected to Home Assistant 😊

Ppl: https://www.buymeacoffee.com/alandtse

[freshfieldx]

This is resolved in https://github.com/alandtse/home-assistant/tree/tesla_oauth_callback. I will be packaging it as a custom component later this week.

That would be really appreciated