Integration not working...again

[400HPMustang]

So after a few days of happiness it looks like Ford has decided to change things up again. Not sure if we can continue to use this integration. Here's the logs

This error originated from a custom integration.

Logger: custom_components.fordpass
Source: helpers/update_coordinator.py:223
Integration: Fordpass (documentation, issues)
First occurred: 9:03:06 AM (1 occurrences)
Last logged: 9:03:06 AM

Error fetching fordpass data: Error communicating with FordPass for 1XXXXXXXXXXXXXXX7

This error originated from a custom integration.

Logger: custom_components.fordpass
Source: custom_components/fordpass/__init__.py:188
Integration: Fordpass (documentation, issues)
First occurred: 9:03:06 AM (1 occurrences)
Last logged: 9:03:06 AM

401 Client Error: Unauthorized for url: https://api.mps.ford.com/api/messagecenter/v3/messages

[tonesto7]

Yeah... ford is restricting the token endpoints further. They have removed support for the grant_type=password
I'm still looking for a solution for my Fordpass Widget... I will post any solutions I find here as well

[itchannel]

I occasionally see that error in my logs but it's more the fordpass API having a wobbly with tokens every so often. Usually fixes itself on the next refresh(15mins). In the last 24hrs it did it twice so at the moment it's not really worth worrying about unless it's stopping permanently for anyone.

@tonesto7 I did notice the latest app update changed the auth, be interesting what you find out, luckily the grant type still works for now on the endpoints my integration requires but for how long who knows.

[tonesto7]

@itchannel I can confirm that requesting a new token (new logins) is broken and ford uses a new sso portal style login.

If you have an existing refresh token it will continue to get a new token. This just affects new logins at the moment.

[Brandawg93]

I'm looking into this as well for my homebridge plugin. I've generated authorization codes before and have code that may be useful to this: https://github.com/Brandawg93/homebridge-nest-cam/blob/master/src/login.ts.

https://github.com/Brandawg93/homebridge-nest-cam/blob/master/src/nest/connection.ts#L77

[tonesto7]

The issue seems to be Ford generates the login service URL per session so there isn't a static login URL anymore. On the ford website the URL is generated here https://www.account.ford.com/etc/fd/fma/bundle.js

[tonesto7]

@itchannel @Brandawg93

With a body of operation=verify&login-form-type=pwd&username=EMAIL_HERE&password=PASSWORD

The identity_source_id param for me always stays the same with 75d08ad1-510f-468a-b69b-5ebc34f773e3 The state_id is what constantly changes

[Brandawg93]

That's the same one I get too. More than likely, the state_id is randomly generated.

[Bert-R]

It's likely to prevent CSRF attacks: https://auth0.com/docs/secure/attack-protection/state-parameters

[tonesto7]

So they do have an official API but do not mention how to request a token. https://developer.ford.com/apis/fordconnect I used the Contact Us section on the developer homepage to see what they say.

[Brandawg93]

They probably have to set the business up to handle Oauth, and then you can use their official page.

[ianjwhite99]

When you his this endpoint

https://sso.ci.ford.com/v1.0/endpoint/default/authorize?redirect_uri=fordapp://userauthorized&response_type=code&scope=openid&max_age=3600&client_id=9fb503e0-715b-47e8-adfd-ad4b7770f73b&code_challenge=EiBNP3VtgKB3ayeBTzAzvQJlXdvY86JeL4f2X49nDW8%3D&code_challenge_method=S256

you are redirected to

https://sso.ci.ford.com/authsvc/mtfim/sps/authsvc?PolicyId=urn:ibm:security:authentication:asf:basicldapuser&identity_source_id=75d08ad1-510f-468a-b69b-5ebc34f773e3&Target=https%3A%2F%2Fsso.ci.ford.com%2Foidc%2Fendpoint%2Fdefault%2Fauthorize%3FqsId%3D3b15437f-47bc-44a4-b1ea-c871c0ed1c86%26client_id%3D9fb503e0-715b-47e8-adfd-ad4b7770f73b#page=login

This is where the state_id and qsId are generated. Likely to prevent CSRF like @Bert-R mentioned.

The qsId can be found in the URL you are redirected to (second URL above), while the stateid can be found in the html of the page (in the image below)_.

Once you submit the login a POST request is made to https://sso.ci.ford.com/authsvc/mtfim/sps/authsvc?identity_source_id=75d08ad1-510f-468a-b69b-5ebc34f773e3&StateId=<state_id> As @tonesto7 mentioned.

Finally, the qsid is used when making the request to the https://sso.ci.ford.com/oidc/endpoint/default/authorize?qsId=<qsId>&client_id=9fb503e0-715b-47e8-adfd-ad4b7770f73b&identity_source_id=75d08ad1-510f-468a-b69b-5ebc34f773e3

That is when the authorization code to fordapp://userauthorized?code=<code> is returned which can be used to get the JWT token.

[tonesto7]

nice find... I can't believe I missed the state_id in the HTML (I was looking for it)

[Brandawg93]

Ya. Just found that info too. That's the same way I had to do it for homebridge-nest-cam. It's a bit hacky, and makes things more difficult for the end-user unfortunately.

[ianjwhite99]

Ford also is validating the cookies as they are coming across to check to make sure the request is actually coming from https://sso.ci.ford.com/. I'm thinking the best solution to this is going to be opening a browser window using selenium, puppeteer or whatever and pulling the cookies & tokens down then using that to make the requests.

It's not going to be the most elegant solution but it might be the only one we have, unfortunately.

[tonesto7]

In my use case, I should be able to encapsulate this all in a nice webview

[tonesto7]

I get the need to improve security. Where there's a will there's a way. If they would just release an official API they would have much more control over those that abuse it. Instead of forcing us to come up with these solutions.

[Brandawg93]

I've done the puppeteer way before. It's pretty horrible.

https://github.com/Brandawg93/homebridge-nest-cam/blob/f40d106620d96972fedb0dd57179d67731cc33aa/src/util/login.ts

[ianjwhite99]

I get the need to improve security. Where there's a will there's a way. If they would just release an official API they would have much more control over those that abuse it. Instead of forcing us to come up with these solutions.

Agreed. It would be nice if they added the ability to generate tokens for user accounts or a client_id and client_secret that could be used with their OAuth.

[tonesto7]

I still don't see how we are going to capture the code... We can't capture fordpass:// URL and we can't change it to a custom URL because it throws a CSIAQ0167E error about invalid redirect_uri

[Brandawg93]

You would need to use something like puppeteer and analyze the response before the redirection and parse the code.

[tonesto7]

Another snafu is that the Code challenge is not a static value.

[Brandawg93]

Here's how you do the code: https://github.com/Brandawg93/homebridge-nest-cam/blob/5d174ba8b11400839313c1eb9d875cf8cdb6d2ec/src/nest/connection.ts#L36

[tonesto7]

The state_id is also in the action attribute in the first URL

[Brandawg93]

This will probably help you but not me...

You can use the auth-token and application id from https://www.ford.com/support/vehicle-dashboard.

That way you can do the auth process completely in the browser.

[tonesto7]

This will probably help you but not me...

You can use the auth-token and application id from https://www.ford.com/support/vehicle-dashboard.

That way you can do the auth process completely in the browser.

Sorry, I'm not following what you mean...

[Brandawg93]

It looks like they haven't properly locked down oauth between applications, so an auth-token generated by the support website will work for the connect apis.

All I did was:

1. log into https://www.ford.com/support/vehicle-dashboard
2. Look in chrome dev console for /users call
3. Copy auth-token and application-id to usapi status call

You would still need a way to intercept the network calls, but this is much easier than trying to figure the redirect_uri stuff with the fordpass app.

[ianjwhite99]

I'm having a hell of a time trying to get the fordapp://userauthorized?code=<code>. I can get all the way to the last request https://sso.ci.ford.com/oidc/endpoint/default/authorize?qsId=<qsId>&client_id=9fb503e0-715b-47e8-adfd-ad4b7770f73b&identity_source_id=75d08ad1-510f-468a-b69b-5ebc34f773e3 using puppeteer but then I get Error: net::ERR_ABORTED.

I'm not sure if its the server blocking me OR if puppeteer is confused because the response is from the fordapp:// URL Scheme.

[tonesto7]

I think i have this almost done with puppeteer... I have to go take care of something but hopefully, I can play with it more tonight.

[itchannel]

Good to see a few people working on it, I agree would be nice if we could just use the official 3rd party API but guessing there's charges associated with it.

[ianjwhite99]

This is a super hacky solution with puppeteer but I figured I'd share.

I was having issues grabbing the fordapp://userauthorized?code=<code>&grant_id=<grant_id>. Puppeteer is confused by the fordapp:// URL Scheme and throws net::ERR_ABORTED.

I attempted to useRequestInterception to try stop the request after successfully hitting the https://sso.ci.ford.com/oidc/endpoint/default/authorize endpoint but I had no luck.

I then however realized that puppeteer logs the reason for the net::ERR_ABORTED to the console. I was able to intercept the message getting logged to the console and grab the fordapp://userauthorized?code=<code>&grant_id=<grant_id> URL scheme successfully using Regex.

[ianjwhite99]

One step closer. Does anybody know what the code_verifier could possibly be?

The initial URL from the FordPass application has the code_challenge and code_challenge_method. But like @tonesto7 stated the value is dynamic.

https://sso.ci.ford.com/v1.0/endpoint/default/authorize?redirect_uri=fordapp://userauthorized&response_type=code&scope=openid&max_age=3600&client_id=9fb503e0-715b-47e8-adfd-ad4b7770f73b&code_challenge=EiBNP3VtgKB3ayeBTzAzvQJlXdvY86JeL4f2X49nDW8%3D&code_challenge_method=S256

[ianjwhite99]

SUCCESS!! I ended up using pkce-challenge and that worked perfectly!

[Brandawg93]

Same! I have this working from beginning to end without puppeteer. I need to clean up the code and then I'll commit it to my repo.

[tonesto7]

@Brandawg93 I'm looking forward to seeing your solution

[Brandawg93]

Ok. Here's what I've got: https://github.com/Brandawg93/homebridge-fordpass/commit/cf6a2babb0ea9183e6e0dd110e814da89991ac52

The only thing that I don't have working is the getVehicles function. Maybe someone else can figure that one out. 😀 I need to put this up for a while and get sleep lol.

[Brandawg93]

And just a forewarning, they will most certainly break this when they add CAPTCHA to their sso UI.

[ianjwhite99]

I've released my changes as well: https://github.com/ianjwhite99/connected-car-node-sdk/commit/a6b5e02be034f3edd09988a327b56fd3fd94de18

My updates are similar to that of @Brandawg93, however I use the same routes that the FordPass application uses instead of going through the redirect_uri=https%3A%2F%2Fwww.ford.com%2Fsupport%2Fvehicle-dashboard. I also grab the Authorization Code from the fordapp://userauthorized? scheme.

Thanks Brandon for figuring out that axios implementation helped me out BIG time!

Edit: I accidentally included puppeteer in that release. It is NOT needed. Removed in the latest version.

[itchannel]

Nice work @Brandawg93 @ianjwhite99 @tonesto7 👍 Just got it working in a very messy form in python. Need to tidy up now :)

[khpylon]

Nice work @Brandawg93 @ianjwhite99 @tonesto7 +1 Just got it working in a very messy form in python. Need to tidy up now :)

@itchannel; I'd love to see your python code so I can figure out what I need to do in Java.

[tonesto7]

Thanks again @Brandawg93 @ianjwhite99 I have this working in a private project again. Now I need to get it working in my Scriptable Script. The main shortcoming is that I can't use Axios or Cookie-Jar.

[Brandawg93]

The main shortcoming is that I can't use Axios or Cookie-Jar.

I was curious how you would be able to do that.

I have it working privately in my plugin again as well. I'm making a few optimizations to my previous code, but it will still include axios and cookie-jar unfortunately. I started playing around with manually getting the cookies from each request and holding them in a variable, but then found cookie-jar.

[tonesto7]

Ok i have it working without cookie-jar now.

You only need to pass the cookies to the next step in 1-4

[Brandawg93]

Hooray for fewer dependencies!

Here's my final code: https://github.com/Brandawg93/homebridge-fordpass/blob/master/src/fordpass-connection.ts

[tonesto7]

I am slowly working through and have removed the dependencies for crypto and base64url as well now.

Once I'm done I will share my code

[tonesto7]

Ugh... i forgot that I can't use Buffer in Scriptable

[itchannel]

Nice work @Brandawg93 @ianjwhite99 @tonesto7 +1 Just got it working in a very messy form in python. Need to tidy up now :)

@itchannel; I'd love to see your python code so I can figure out what I need to do in Java.

@khpylon check out branch 1.34 for how I've implemented it in python. I'm already using session handling in requests so that handles cookies etc for the multiple calls. It's still a little messy but should give you a guide.

[ianjwhite99]

Thanks @itchannel for sharing your python release. It was a big help in getting the auth working as quickly as possible for my python SDK! https://github.com/ianjwhite99/connected-car-python-sdk/commit/e4a23f9de98d5e78a4a7f4ccf656c7061b01d572

[tonesto7]

@itchannel @Brandawg93 @ianjwhite99

Do you guys got any ideas on vanilla code to simulate this line?: const hashBuf = Buffer.from(hash, "hex");

Here is the hash code output and the string output of the code above.

hash 0b2a323431340024dd29c574fa25f4bae5a09b46d459de57bd8339ff8542792a

hashBuf: <Buffer 0b 2a 32 34 31 34 00 24 dd 29 c5 74 fa 25 f4 ba e5 a0 9b 46 d4 59 de 57 bd 83 39 ff 85 42 79 2a>

[ianjwhite99]

@itchannel @Brandawg93 @ianjwhite99

Do you guys got any ideas on vanilla code to simulate this line?: const hashBuf = Buffer.from(hash, "hex");

Here is the hash code output and the string output of the code above.

hash 0b2a323431340024dd29c574fa25f4bae5a09b46d459de57bd8339ff8542792a

hashBuf: <Buffer 0b 2a 32 34 31 34 00 24 dd 29 c5 74 fa 25 f4 ba e5 a0 9b 46 d4 59 de 57 bd 83 39 ff 85 42 79 2a>

Someone correct me if I'm wrong but...

I don't believe you can use vanilla code to simulate that with Node. The Buffer class in Node corresponds to some raw memory allocated outside V8. Meaning you would have to interact with the raw memory to simulate the Buffer.from() function.

[rene-bayer]

@itchannel do you want to release it for hacs? :)