MyQ API Update.

[hjdhjd]

@arraylabs I'm the developer of homebridge-myq - here's the snippet you'll need to update for pymyq...

https://github.com/hjdhjd/homebridge-myq/commit/de22f7ed21ee4a7465cea54e6ccc1c70bc101138

Best of luck!

[ehendrix23]

I just pushed Beta 5. In manifest.json for myq custom component set requirements to: "requirements": ["git+https://github.com/arraylabs/pymyq@3.0.0b5#pymyq==3.0.0b5", "pkce==1.0.2"],

I also made some changes to the myq component which will:

1. Change state to opening or closing in HASS when cover is being opened or closed
2. More frequently poll when state is being changed from HASS (every 5 seconds) and thus able to provide state faster back to HASS
3. Update is now done every 15 seconds instead of every 30 seconds

So please also update you custom myq component from: https://github.com/ehendrix23/home-assistant/tree/Bump_MyQ_to_3.0.0/homeassistant/components/myq

Thx.

[ehendrix23]

If I may suggest a nuance to polling:

homebridge-myq polls every 12 seconds, by default. I’d suggest 12-15 seconds as a reasonable default. It also increases the polling resolution to every 3-5 seconds when it detects a state change. The argument being that as state changes are occurring, you should poll more to give people that “realtime” feeling. 😄 After a defined interval say 120-180 seconds (it’s actually longer in homebridge-myq, but I’ve been thinking of changing it in the near future) of no state changes, it returns back to it’s default polling interval.

You might want to consider something along these lines...it provides most of the benefits of both worlds, in practice, given that most folks aren’t constantly opening and closing their garage doors. 😄

The task to update the information (polling) is initiated from Home Assistant itself and is put on there with a fixed interval. To do something like this I would need to run my own loop instead to poll and then inform Home Assistant when a state changed. Can be done, but I would say not in scope right now.

Based on your suggestion I did make a change where if the garage is being opened/closed from Home Assistant then it will be retrieving it every 5 seconds. Thus at least when being opened/closed from Home Assistant it would be closed to this "real" feel.

[bdraco]

@ehendrix23

I wanted to proactively point out this before it gets to Home Assistant review, to avoid any disappointment or hold ups.

The html parsing in https://github.com/arraylabs/pymyq/blob/v6_API/pymyq/api.py#L38 is going to be an issue with ADR0004 (https://github.com/home-assistant/architecture/blob/master/adr/0004-webscraping.md)

Tesla's recent api changes require something similar which is holding up https://github.com/home-assistant/core/pull/45766 From what I've gathered, alandtse is looking into a way to proxy and capture the the oauth callback to avoid the web scraping since to similar myq, tesla captures the callback into their app. It might save some time to collaborate with him on this problem.

[ehendrix23]

@ehendrix23

I wanted to proactively point out this before it gets to Home Assistant review, to avoid any disappointment or hold ups.

The html parsing in https://github.com/arraylabs/pymyq/blob/v6_API/pymyq/api.py#L38 is going to be an issue with ADR0004 (https://github.com/home-assistant/architecture/blob/master/adr/0004-webscraping.md)

Tesla's recent api changes require something similar which is holding up home-assistant/core#45766 From what I've gathered, alandtse is looking into a way to proxy and capture the the oauth callback to avoid the web scraping since to similar myq, tesla captures the callback into their app. It might save some time to collaborate with him on this problem.

Hmmmm. Then this might be an issue. We scrape the returned web page to get the value for __RequestVerificationToken which is required in the response for login. Right now there is also no method yet to refresh the token except for doing a re-authentication.

Will be following that but if web scraping not allowed then it seems MyQ will be going out of Home Assistant and might need to be added to HACS instead.

Where did you see that he is looking into a way to proxy and capture the oauth callback to avoid web scraping?

[alandtse]

Where did you see that he is looking into a way to proxy and capture the oauth callback to avoid web scraping?

I discussed it in discord. I do that with Alexa Media Player and was asking whether I should break it out as a library in general to solve Tesla and other oauth situations where we don't have third party access.

[hjdhjd]

There are two problems:

1. In general, refreshing the token isn’t worthwhile, given the login process is automated. Arguably, it’s only true value is to not have users interact with a login UI more than is absolutely necessary. Given that’s a solved problem...refreshing the token is of marginal value.
2. Even when I tried to refresh the tokens, it seems the myQ team didn’t even bother to implement their end of the refresh endpoints. :smile: Their own app executes the full login process when needed.

You have to remember: myQ is not a realtime API, nor intended to be. It is used infrequently, at best, to check status. The app, as intended, logs in to check status and alerts you. Nobody spends all day in the app (in theory). Why would you implement refresh for something that’s used for 30 seconds to 5 minutes at a time, max, ever?

How we collectively simulate a “realtime-like” experience is, to say the least, an unsupported use of the API’s intent and design by creating longer-lived sessions than they ever intended. If the API was well-designed for that realtime experience, we’d have some way to register callbacks / webhooks / SOMETHING to get notified of a state change rather than having to poll for it constantly.

That alone should tell anyone that this API was never designed to do what we have done with it. That it works at all, is a testament to our collective stubbornness.

Native myQ-related push notifications happen exclusively on the myQ server end and are not ever (to my knowledge) exposed to the outside world, including their native apps. They work because the myQ servers have their own APNS (and whatever Android uses) that they trigger directly.

Put another way: myQ has outsourced realtime notifications to Apple and Google, effectively. Why? Because it’s cheaper and makes for a less complex API and fewer edge cases to deal with. When all you need to do is worry about sending Apple or Google a push notification and make it their respective problem to deliver them, you don’t need to worry about a realtime API. :smile:

Hope this helps shed a little light on this.

[ehendrix23]

I just pushed Beta 6. In manifest.json for myq custom component set requirements to: "requirements": ["git+https://github.com/arraylabs/pymyq@3.0.0b6#pymyq==3.0.0b6", "pkce==1.0.2", "beautifulsoup4==4.9.3"],

Main change here is how we get the values from the login page to submit back to for login. Any hidden fields are now automatically returned back. Further we check for type "email" and type "password" and then use the name for that field as the name to return back.

[ehendrix23]

Where did you see that he is looking into a way to proxy and capture the oauth callback to avoid web scraping?

I discussed it in discord. I do that with Alexa Media Player and was asking whether I should break it out as a library in general to solve Tesla and other oauth situations where we don't have third party access.

In Alexa Media Player do you refresh token using token refresh after initial authentication? Once initial login is done (i.e. during component setup) do you ever need to do this again using userid/password? If so, do you then store the fields and what needs to be returned somewhere? What if that then changes? Require the user to re-authenticate themselves again?

Only method for us to refresh the token is by going back through the same process to re-authenticate.

[alandtse]

In Alexa Media Player do you refresh token using token refresh after initial authentication? Once initial login is done (i.e. during component setup) do you ever need to do this again using userid/password? If so, do you then store the fields and what needs to be returned somewhere? What if that then changes? Require the user to re-authenticate themselves again?

Only method for us to refresh the token is by going back through the same process to re-authenticate.

We do all of the above. AMP has been dealing with different auth login issues for a long time so we've implemented multiple log in methods and automatic reauth. This means we save cookies, tokens, usernames, passwords, and even have a otp generator. It only pops up a reauth dialogue if auto reauth fails and then saves the new data on success.

The oauth proxy captures everything except for the otp generator as that is a separate setting.

[dseven]

Installed beta 6 on home assistant 2021.2.0 yesterday. Overnight, it logged:

2021-02-04 01:08:19 ERROR (MainThread) [custom_components.myq] Error fetching myq devices data: Error requesting data from https://accounts.myq-cloud.com/api/v6.0/accounts: Cannot connect to host accounts.myq-cloud.com:443 ssl:default [Try again]

The home assistant device was "unavailable" from 01:08:19 until 01:16:02. It seems to be working fine now (without any intervention).

FWIW, my "production" home assistant instance, which is still running the old component, didn't report any issues or unavailability.

[ehendrix23]

Installed beta 6 on home assistant 2021.2.0 yesterday. Overnight, it logged:

2021-02-04 01:08:19 ERROR (MainThread) [custom_components.myq] Error fetching myq devices data: Error requesting data from https://accounts.myq-cloud.com/api/v6.0/accounts: Cannot connect to host accounts.myq-cloud.com:443 ssl:default [Try again]

The home assistant device was "unavailable" from 01:08:19 until 01:16:02. It seems to be working fine now (without any intervention).

FWIW, my "production" home assistant instance, which is still running the old component, didn't report any issues or unavailability.

I did not see errors on my end but based on what is reported I would say most likely an issue on MyQ's end. V6 did not even change anything in that portion of the coding. The old component is connecting to another MyQ URL then the new V6 API.

[hjdhjd]

There are still going to be occasional myQ failures, even with v6. Yeah, they moved to Azure. Yeah, they updated some things. But it’s still myQ. :smile: Expect occasional failures.

For what it’s worth...I’ve seen a ~80% reduction in random endpoint failures / resets, but that’s anecdotal over the past month or so.

[ehendrix23]

There are still going to be occasional myQ failures, even with v6. Yeah, they moved to Azure. Yeah, they updated some things. But it’s still myQ. 😄 Expect occasional failures.

For what it’s worth...I’ve seen a ~80% reduction in random endpoint failures / resets, but that’s anecdotal over the past month or so.

Yeah, once I changed the logic for authentication so it will close the HTTPS connection after authentication is complete I have not even seen the connection reset by peer messages anymore.

[ehendrix23]

v3.0.0 has been released. Submitted PR https://github.com/home-assistant/core/pull/46079 to get HASS updated to new version.

Closing this issue now.

[hjdhjd]

Congrats @ehendrix23 - here’s hoping we can decode more of the v6 landscape. There are tantalizing things that are now accessible to us like cameras and locks...it’s just going to take time (and motivation) to take them on. If you ever visit the Homebridge Discord...drop me a line. Best of luck.

[dseven]

Much thanks to @ehendrix23 and @hjdhjd for all the hard work on this, and to the other testers too!

[ehendrix23]

Congrats @ehendrix23 - here’s hoping we can decode more of the v6 landscape. There are tantalizing things that are now accessible to us like cameras and locks...it’s just going to take time (and motivation) to take them on. If you ever visit the Homebridge Discord...drop me a line. Best of luck.

Just replied to your comment there. :-) And put a watch on your repository so when you do updates I'll get notified .

[ehendrix23]

FYI,

PR home-assistant/core#46079 is still under review due to violation ADDR004. This requirement might get adjusted.

In the mean time the myQ repository is also available through HACS as a custom component. See: https://github.com/ehendrix23/hass_myq

[GaryOkie]

Wow!!! PR got approved to merge and the ADR was amended to allow auth scraping! Congratulations Erik - just incredible.

Home Assistant Release 2021.2.3 - February 11

• Bump pymyq to 3.0.1 (@ehendrix23 - #46079) (myq docs)