MyQ API Update. - Githubissues

hjdhjd commented 3 years ago

@arraylabs I'm the developer of homebridge-myq - here's the snippet you'll need to update for pymyq...

https://github.com/hjdhjd/homebridge-myq/commit/de22f7ed21ee4a7465cea54e6ccc1c70bc101138

Best of luck!

ehendrix23 commented 3 years ago

Thx!! Looked at the changes and it seems that we were pretty much were already using those URLs for a while. Mine and others are currently working. Not sure if this was from previously and never updated resulting in it still working or not. :-)

Am keeping an eye on your repository for V6 so that I can start working on that for pymyq!

Thanks for all the work and reverse engineering. That is a great help for us all!

hjdhjd commented 3 years ago

Reopening this one @ehendrix23

https://github.com/hjdhjd/homebridge-myq/releases/tag/v2.6.1

Happy to answer questions and be helpful. Particularly look at the login process...that's where all the magic is. The v6 API is primarily used for login, and the rest of the API has changed, but resembles the previous one in many ways. Some things like the devices JSON are the same, but others, like opening and closing devices have changed more.

Code is, as always, well commented.

Enjoy...happy to share the knowledge and move our community forward...would appreciate an acknowledgement , but not necessary...just a lot of work in my free time lately to get this going. 😄 Cheers.

ehendrix23 commented 3 years ago

Thx @hjdhjd !!!! Started to work on this yesterday by analyzing your coding to understand what is being done and slowly updating pymyq. So far no questions. :-)

Thank you again for all your work to reverse engineer the new API and sharing it with us. An acknowledgment is the least we can do. :-)

ehendrix23 commented 3 years ago

Test version for MyQ using V6 API is ready. This version also includes support for lamps.

For use with HASS:

Create a custom component for myq in HASS (create folder custom_components/myq in your hass config folder)
Retrieve the files from my branch: https://github.com/ehendrix23/home-assistant/tree/Bump_MyQ_to_3.0.0/homeassistant/components/myq
Put those files in the custom component folder you created in step 1
Restart HASS

Please report any issues here.

Updated pymyq is available here: https://github.com/arraylabs/pymyq/tree/v6_API/pymyq HASS will install automatically.

FlyingDiver commented 3 years ago

That git link is bad. Maybe it's not public?

ehendrix23 commented 3 years ago

That git link is bad. Maybe it's not public?

Updated URL.

FlyingDiver commented 3 years ago

Probably ought to be: https://github.com/arraylabs/pymyq/tree/v6_API

hjdhjd commented 3 years ago

@ehendrix23 Support for lamps is incomplete in my plugin...did you have a lamp to test with to verify it works? :smile:

ehendrix23 commented 3 years ago

@hjdhjd No I have not, I do not have a lamp. Hoping someone else then does and can provide result. :-) Thought your implementation was complete. :-)

hjdhjd commented 3 years ago

@ehendrix23 It was an educated guess. We should have access to one in the coming days and hope to get that completed within a week or two. It’s just a function of getting the right endpoint sniffed.

brbeaird commented 3 years ago

I do have a lamp device that has been working with V5 (just started working through the V6 changes). For what it's worth, the V5 on/off commands are "turnon" and "turnoff"

garyak commented 3 years ago

Followed the instructions above. I did not remove existing MyQ integration. All the latest HA code.

Logger: homeassistant.setup Source: setup.py:138 First occurred: 9:01:32 AM (1 occurrences) Last logged: 9:01:32 AM Setup failed for myq: Unable to import component: No module named 'pkce'

2021-01-29 09:02:33 ERROR (MainThread) [homeassistant.config_entries] Error occurred loading configuration flow for integration myq: No module named 'pkce'
2021-01-29 09:02:33 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 535, in async_create_flow
integration.get_platform("config_flow")
File "/usr/src/homeassistant/homeassistant/loader.py", line 424, in get_platform
cache[full_name] = self._import_platform(platform_name)
File "/usr/src/homeassistant/homeassistant/loader.py", line 429, in _import_platform
return importlib.import_module(f"{self.pkg_path}.{platform_name}")
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/config/custom_components/myq/__init__.py", line 6, in <module>
import pymyq
File "/usr/local/lib/python3.8/site-packages/pymyq/__init__.py", line 2, in <module>
from .api import login # noqa
File "/usr/local/lib/python3.8/site-packages/pymyq/api.py", line 11, in <module>
from pkce import generate_code_verifier, get_code_challenge
ModuleNotFoundError: No module named 'pkce'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 119, in async_init
flow = await self.async_create_flow(handler, context=context, data=data)
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 542, in async_create_flow
raise data_entry_flow.UnknownHandler
homeassistant.data_entry_flow.UnknownHandler

hjdhjd commented 3 years ago

I do have a lamp device that has been working with V5 (just started working through the V6 changes). For what it's worth, the V5 on/off commands are "turnon" and "turnoff"

That's not the issue. :smile: The issue is endpoints, not commands. We've all had our plugins working in v5 for lamps. v6 is a new animal.

ehendrix23 commented 3 years ago

@garyak

Re-download manifest.json, added pkce as a requirement. Won't be an issue once released. :-)

garyak commented 3 years ago

Up and running.

garyak commented 3 years ago

Recurring errors:

2021-01-29 10:36:42 ERROR (MainThread) [pymyq.api] Error trying to re-authenticate to myQ service: Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer
2021-01-29 10:36:42 ERROR (MainThread) [custom_components.myq] Error fetching myq devices data: Error trying to re-authenticate to myQ service: Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer

GaryOkie commented 3 years ago

Update installed on Home Assistant Core 2021.5 using custom_components/myq folder. It's not working for me. Restarted with pymyq: debug enabled. Log follows:

EDIT: Commented too soon. It eventually did connect and start working! Not clear why it took so many attempts to connect.

Second EDIT: I was remiss is not first profusely thanking you @ehendrix23 for all your work on pymyq lately and getting this new new authentication method operational. And of course a big shout out to @hjdhjd for the huge contribution to sharing their Homebridge code that solved this bugger. I'll continue to assist with testing and hope these initial connection failures are just a sporadic thing that takes a minute to settle down. (I do have 2 openers that are being managed, and one is another location),

2021-01-29 11:49:40 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for myq which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant.

2021-01-29 11:50:12 DEBUG (MainThread) [pymyq.api] Performing initial authentication into MyQ
2021-01-29 11:50:12 DEBUG (MainThread) [pymyq.api] Initiating OAuth authentication
2021-01-29 11:50:12 DEBUG (MainThread) [pymyq.api] Retrieving authentication page
2021-01-29 11:50:12 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/authorize and headers {'redirect': 'follow'}

2021-01-29 11:50:13 DEBUG (MainThread) [pymyq.request] Attempt 1 request failed with exception:: [Errno 104] Connection reset by peer
2021-01-29 11:50:13 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer"; trying again in 2 seconds
2021-01-29 11:50:15 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/authorize and headers {'redirect': 'follow'}
2021-01-29 11:50:15 DEBUG (MainThread) [pymyq.request] Attempt 2 request failed with exception:: [Errno 104] Connection reset by peer
2021-01-29 11:50:15 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer"; trying again in 4 seconds
2021-01-29 11:50:19 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/authorize and headers {'redirect': 'follow'}
2021-01-29 11:50:20 DEBUG (MainThread) [pymyq.request] Attempt 3 request failed with exception:: [Errno 104] Connection reset by peer
2021-01-29 11:50:20 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer"; trying again in 5 seconds
2021-01-29 11:50:25 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/authorize and headers {'redirect': 'follow'}
2021-01-29 11:50:25 DEBUG (MainThread) [pymyq.request] Attempt 4 request failed with exception:: [Errno 104] Connection reset by peer
2021-01-29 11:50:25 ERROR (MainThread) [pymyq.api] Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer
2021-01-29 11:50:25 WARNING (MainThread) [homeassistant.config_entries] Config entry for myq not ready yet. Retrying in 5 seconds

2021-01-29 11:50:30 DEBUG (MainThread) [pymyq.api] Performing initial authentication into MyQ
2021-01-29 11:50:30 DEBUG (MainThread) [pymyq.api] Initiating OAuth authentication
2021-01-29 11:50:30 DEBUG (MainThread) [pymyq.api] Retrieving authentication page
2021-01-29 11:50:30 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/authorize and headers {'redirect': 'follow'}
2021-01-29 11:50:30 DEBUG (MainThread) [pymyq.request] Attempt 1 request failed with exception:: [Errno 104] Connection reset by peer
2021-01-29 11:50:30 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer"; trying again in 2 seconds
2021-01-29 11:50:32 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/authorize and headers {'redirect': 'follow'}
2021-01-29 11:50:32 DEBUG (MainThread) [pymyq.request] Attempt 2 request failed with exception:: [Errno 104] Connection reset by peer
2021-01-29 11:50:32 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer"; trying again in 4 seconds
2021-01-29 11:50:36 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/authorize and headers {'redirect': 'follow'}
2021-01-29 11:50:37 DEBUG (MainThread) [pymyq.request] Attempt 3 request failed with exception:: [Errno 104] Connection reset by peer
2021-01-29 11:50:37 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer"; trying again in 5 seconds
2021-01-29 11:50:42 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/authorize and headers {'redirect': 'follow'}
2021-01-29 11:50:42 DEBUG (MainThread) [pymyq.request] Attempt 4 request failed with exception:: [Errno 104] Connection reset by peer
2021-01-29 11:50:42 ERROR (MainThread) [pymyq.api] Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer
2021-01-29 11:50:42 WARNING (MainThread) [homeassistant.config_entries] Config entry for myq not ready yet. Retrying in 10 seconds

and... a successful connection:

2021-01-29 11:50:52 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer"; trying again in 2 seconds
2021-01-29 11:50:54 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/authorize and headers {'redirect': 'follow'}
2021-01-29 11:50:55 DEBUG (MainThread) [pymyq.request] Response:
2021-01-29 11:50:55 DEBUG (MainThread) [pymyq.request] Response Code: 200
2021-01-29 11:50:55 DEBUG (MainThread) [pymyq.request] Headers: ((b'Cache-Control', b'no-cache, no-store'), (b'Pragma', b'no-cache'), (b'Content-Type', b'text/html; charset=utf-8'), (b'Content-Encoding', b'gzip'), (b'Vary', b'Accept-Encoding'), (b'Request-Context', b'appId=cid-v1:4dxxxxxx'), (b'api-supported-versions', b'1.0'), (b'X-Content-Type-Options', b'nosniff'), (b'X-Frame-Options', b'SAMEORIGIN'), (b'Referrer-Policy', b'no-referrer'), (b'MyQ-CorrelationId', b'xxx-xxx-xxx2'), (b'Date', b'Fri, 29 Jan 2021 17:50:54 GMT'), (b'Content-Length', b'1476'), (b'Set-Cookie', b'AntiForgery.MyQ.Identity=CfDJ8Exxx; path=/; secure; samesite=strict; httponly'), (b'Set-Cookie', b'TS01xxxxxxd6xxxxx954d; Path=/; Domain=.partner-identity.myq-cloud.com'))

and a whole lot more debug stuff follows. (mainly because pymyq requests are polling every 30 seconds)

hjdhjd commented 3 years ago

@ehendrix23 When I was working through all this in decoding v6 for homebridge-myq here are a couple of thoughts based on the above:

Definitely exclude the user agent header. In my plugin, I set the UA to null because node-fetch will interpret that to not send a UA header to myQ. Not sure how that works in the python / HA world.
Connection resets will occasionally happen...that's just life with the myQ API. It happens a lot less frequently in v6, once you're up and running, in my experience.
The errors on redirect above...that's where all the complexity and magic is in my plugin. myQ v6 uses OAuth + PKCE for the login process. You can search online what that workflow looks like (generically), but the important element here is: myQ hands back cookies that have things appended to them that tend to muck up the process. That's why trimSetCookie exists. :smile: You'll also see that I explicitly interrupt the redirects in order to cleanup the cookies before passing them along to complete the OAuth login process. It's messy, but once you get your head around why it's needed, it makes a lot more sense.
The TTL for the OAuth keys is an hour. You'll need to acquire new keys every hour.

You may have already gathered most of this in reading through my code, but in case it was unclear, I wanted to provide some more context to help you guys along.

ehendrix23 commented 3 years ago

@hjdhhjd

Your coding was easy to read. I did have to figure out some things on what exactly it is from a coding language. :-)

We set a skip_auto_headers for user agent to prevent it from being set and then don't provide one.
Yeah, seen a few of them. I just finished some coding allowing us to refresh the token independent of commands being sent or getting device status. So having some of these connection resets during authentication should not impact so much. Coding was already in place to retry any requests to MyQ for 5 times each time waiting a bit longer before sending it
Yep, saw all that and did exact same on my end. It also helped as I went through 1 request to see what was returned and then use that in combination with your coding to determine what had to be done. Doing the trimming as well. :-)
I'm using the expires_in and then dividing that by 2 to start re-authentication for a new token. Figured (in combination with it being able to re-authenticate independent of refreshing) that gives sufficient time to get through. I also have a default set to 10 minutes in case expires_in is not returned or is like 0. Not sure if MyQ uses the expires. For testing I had it wait for 7200 seconds and token was still valid.

Thanks again for EVERYTHING you have done. I would not have been able to update without all the work you did.

ehendrix23 commented 3 years ago

I just completed some more updates. To ensure you get the latest version in HASS, updated manifest.json and change 3.0.0 to 3.0.0b3. Or just redownload the manifest.json from https://github.com/ehendrix23/home-assistant/tree/Bump_MyQ_to_3.0.0/homeassistant/components/myq.

Biggest change now is that trying to update the token will not interfere with getting device status updates or sending commands. We start trying to get a new token at half the expire time for the token (current expire is 1 hour, hence at 30 minutes) as a separate task. That gives us 30 minutes to get a new token before existing actually expires.

Further, should the token be expired when sending a request (currently assuming it will return 401) then a new authentication is done and if successful the request is re-processed. Thus if it should happen upon sending a command the command will not fail.

Let me know how it goes. If anyone has a lamp, please check if it works or not. Support is not in HASS but you can try the example.py (email and password needs to be entered in there).

Thx.

ehendrix23 commented 3 years ago

Recurring errors:

2021-01-29 10:36:42 ERROR (MainThread) [pymyq.api] Error trying to re-authenticate to myQ service: Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer
2021-01-29 10:36:42 ERROR (MainThread) [custom_components.myq] Error fetching myq devices data: Error trying to re-authenticate to myQ service: Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer

Sadly enough this sometimes happens and it mainly seems to be on authentication page. I just did an update so that authentication is run as a separate task instead allowing device updates etc. to still occur. See my post from a few minutes ago.

ehendrix23 commented 3 years ago

I do have a lamp device that has been working with V5 (just started working through the V6 changes). For what it's worth, the V5 on/off commands are "turnon" and "turnoff"

Maybe you can try it with example.py to see if what we have works or not? :-)

hjdhjd commented 3 years ago

@hjdhhjd

Your coding was easy to read. I did have to figure out some things on what exactly it is from a coding language. :-)

We set a skip_auto_headers for user agent to prevent it from being set and then don't provide one.

Yeah, seen a few of them. I just finished some coding allowing us to refresh the token independent of commands being sent or getting device status. So having some of these connection resets during authentication should not impact so much. Coding was already in place to retry any requests to MyQ for 5 times each time waiting a bit longer before sending it

Yep, saw all that and did exact same on my end. It also helped as I went through 1 request to see what was returned and then use that in combination with your coding to determine what had to be done. Doing the trimming as well. :-)

I'm using the expires_in and then dividing that by 2 to start re-authentication for a new token. Figured (in combination with it being able to re-authenticate independent of refreshing) that gives sufficient time to get through. I also have a default set to 10 minutes in case expires_in is not returned or is like 0. Not sure if MyQ uses the expires. For testing I had it wait for 7200 seconds and token was still valid.

Thanks again for EVERYTHING you have done. I would not have been able to update without all the work you did.

Re: trimming...sorry about that. I couldn't help but get a little fancy with the code there. I wanted to see if I could make an elegant one-liner I could quickly get my head around there. :smile:

In my testing, myQ doesn't seem to use the expires. The whole thing is a little funky...but once it's going it makes for a better experience all around I hope.

And you're welcome...happy to help contribute knowledge to the cause. I'm sure it's only a matter of time before we have to figure this out all over again for the next major API revision...

On lamps...hoping to have that sorted out properly in the next few days. One thing at a time...

Oh...did you catch how the accounts JSON works now and how this is the vector that enables guest access? That was last weekend's update that I pushed.

GaryOkie commented 3 years ago

Updated to 3.0b3 - looking good so far! Previous version filled log with bunches of python errors every 30 seconds.

So at startup, I am seeing several warnings and 1 error, which aren't ideal to be showing up, especially the error. Either the updated doc should mention that these messages need to be ignored or see if they can be tidied up some more. I don't know - maybe making all the retrying messages DEBUG or INFO-level so they are normally hidden and only show an error after x number of failed attempts?

2021-01-29 20:26:46 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer" (attempt #1/5)"; trying again in 2 seconds
2021-01-29 20:26:49 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer" (attempt #2/5)"; trying again in 4 seconds
2021-01-29 20:26:53 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer" (attempt #3/5)"; trying again in 5 seconds
2021-01-29 20:26:58 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer" (attempt #4/5)"; trying again in 5 seconds

2021-01-29 20:27:04 ERROR (MainThread) [pymyq.api] Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer

2021-01-29 20:27:04 WARNING (MainThread) [homeassistant.config_entries] Config entry for myq not ready yet. Retrying in 5 seconds
2021-01-29 20:27:09 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer" (attempt #1/5)"; trying again in 2 seconds
2021-01-29 20:27:11 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer" (attempt #2/5)"; trying again in 4 seconds
2021-01-29 20:27:16 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer" (attempt #3/5)"; trying again in 5 seconds

garyak commented 3 years ago

Updated yesterday afternoon. Seeing groups of six repetitive errors like this:

2021-01-30 05:21:01 ERROR (MainThread) [pymyq.api] Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer
2021-01-30 05:21:15 ERROR (MainThread) [pymyq.api] Error trying to re-authenticate to myQ service: Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer
2021-01-30 05:21:32 ERROR (MainThread) [pymyq.api] Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer
2021-01-30 05:21:45 ERROR (MainThread) [pymyq.api] Error trying to re-authenticate to myQ service: Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer
2021-01-30 05:22:02 ERROR (MainThread) [pymyq.api] Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer
2021-01-30 05:22:16 ERROR (MainThread) [pymyq.api] Error trying to re-authenticate to myQ service: Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer

I assume this reflects your retries, Logging is set to error, so I'm not catching warnings. History indicates the binary sensor remains connected during these events. Previous history showed connect/disconnect cycles every few hours. The custom_components.myq error, from an earlier post, no longer appears in the log.

GaryOkie commented 3 years ago

I concur with Garyak's findings. My comment "looking good" was meant to refer to what appears to be a stable connection from the front-end and being able to open/close. The profuse number of warnings and errors is not a good look!

Since my previous post, there are 1400 Warnings ("request failed, errno104, retrying"), and 683 pymyq.api ERRORs as shown above.

ds7771 commented 3 years ago

Is there a recommended method to install this code in Home Assistsnt core on docker? I am not running HASS. If so, happy to help test.

GaryOkie commented 3 years ago

@ds7771 - Unless you are planning on helping with testing a lamp module, I would hold off for a bit to see if these abundant error messages can be tamed (other than upping the pymyq logging level).

You can download Erik's updated HA myq via DownGit using this link:
https://downgit.github.io/#/home?url=https://github.com/ehendrix23/home-assistant/tree/Bump_MyQ_to_3.0.0/homeassistant/components/myq

Extracting zip creates a myq folder with the V6 code. Copy myq folder to HA Core's /config/custom_components folder. Restart HA. That's it.

dseven commented 3 years ago

FWIW, similar results here (on a test instance of homeassistant 2021.1.5:

homeassistant_dev    | 2021-01-30 11:07:05 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for myq which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant.
homeassistant_dev    | 2021-01-30 11:07:23 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer" (attempt #1/5)"; trying again in 2 seconds
homeassistant_dev    | 2021-01-30 11:37:30 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer" (attempt #1/5)"; trying again in 2 seconds
homeassistant_dev    | 2021-01-30 11:37:32 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer" (attempt #2/5)"; trying again in 4 seconds
homeassistant_dev    | 2021-01-30 11:37:36 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer" (attempt #3/5)"; trying again in 5 seconds
homeassistant_dev    | 2021-01-30 12:01:28 WARNING (MainThread) [pymyq.request] Request failed with " [Errno 104] Connection reset by peer" (attempt #1/5)"; trying again in 2 seconds

The second reset occurred 30 mins after startup, which makes me wonder if there's just some sort of maximum session time on the server side, although the third reset was only 24 mins after the second, and it wouldn't account for the reset immediately after startup. Will leave it running for a bit and see what happens.

Otherwise, it opened my garage door OK. I don't have lights or anything else fancy.

dseven commented 3 years ago

After running for nearly 24 hours, I'm definitely seeing a pattern of these resets happening on roughly 30 minute intervals. It doesn't happen (get logged) every 30 mins, but often - e.g. I see it at 01:48, 02:48, 03:20, 03:51, 04:51, 05:22, 05:52, 06:23,

So far, home assistant doesn't seem to be bothered about it - no unavailability evident in the history.

ehendrix23 commented 3 years ago

@dseven, currently token is set to expire every hour. we take half of that (30 minutes) to start trying to get a new token. This to ensure we get an updated token before it expires (and thus not interfere with ability to get information etc). That is why you see it happening about every 30 minutes.

ehendrix23 commented 3 years ago

Some more updates, mainly around logging (switched to debug from warning or error) and setting it so that error messages related to authentication are only produced when it impacts getting device information (i.e. token is now expired). Also, if username/password provided is wrong then we ensure we don't keep on authenticating with that.

To ensure you get the latest version in HASS, updated manifest.json and change 3.0.0b3 to 3.0.0b4 . Or just redownload the manifest.json from https://github.com/ehendrix23/home-assistant/tree/Bump_MyQ_to_3.0.0/homeassistant/components/myq.

Thx.

hjdhjd commented 3 years ago

@ehendrix23 Take a look at the latest homebridge-myq - got lamps fully working with the new API. My educated guess was pretty close to begin with. :smile: Should’ve just trusted my instincts instead of thinking they’d keep more of the legacy commands. 😉

dseven commented 3 years ago

@dseven, currently token is set to expire every hour. we take half of that (30 minutes) to start trying to get a new token. This to ensure we get an updated token before it expires (and thus not interfere with ability to get information etc). That is why you see it happening about every 30 minutes.

That makes me a bit concerned... it sounds like the connections are getting dropped on the server side, but we don't notice until we attempt token refresh. Does this mean that we may not get status updates (e.g. garage door was opened using the clicker) because we're happily listening on a connection that we don't know is already dead? Maybe we need some sort of periodic keepalive....

hjdhjd commented 3 years ago

@dseven, currently token is set to expire every hour. we take half of that (30 minutes) to start trying to get a new token. This to ensure we get an updated token before it expires (and thus not interfere with ability to get information etc). That is why you see it happening about every 30 minutes.

That makes me a bit concerned... it sounds like the connections are getting dropped on the server side, but we don't notice until we attempt token refresh. Does this mean that we may not get status updates (e.g. garage door was opened using the clicker) because we're happily listening on a connection that we don't know is already dead? Maybe we need some sort of periodic keepalive....

I don’t know the specifics of this implementation, but as someone who spends a lot of time understanding the myQ API, and documenting it so that other projects can build on my work, here’s what I can tell you: you never get periodic status updates from the myQ API. That’s not how it works.

The myQ API provides a status update when queried. That’s it. You can choose to query it every ten seconds, or every two seconds, but there’s no status being pushed from the API to our respective plugins. How accurate the current status is depends on the “resolution” or granularity of how frequently you request the current status.

In my plugin, for instance, if a status update fails, we abandon the current authentication token, re-login, get a new one, and retry at the next interval. An alternative would be to retry again immediately after getting a new authentication token, but that creates it’s own potential problems in the form of infinite loops, etc. There’s no one right answer here...it’s up to each developer to decide what makes sense in their codebase.

If there’s an underlying flaw here it’s the myQ API’s inability to provide some sort of a publish / subscribe capability to allow updates to be pushed to us, rather than have to constantly query for state changes.

And finally...there’s the question of blacklisting and abuse. If you query the API too often (say every second or two, all the time), you have a decent probability of having your account locked or terminated, because the myQ team isn’t crazy about having a lot of people querying their API every seconda or two and using it to create near-realtime updates. There’s a happy medium we all try to hit...but it’s an imperfect solution to an imperfect API.

dseven commented 3 years ago

Thanks @hjdhjd - that all makes sense, and I do understand that we're walking on eggshells a bit with all of this. I guess I need to dig in a bit to understand how the polling is being done. It seems like polling would be susceptible to connection reset in the same way as token refreshes, but maybe there's more subtlety to it.

I actually half gave up on my myQ garage door opener, as it was taking too long for home assistant to notice that the door was opening/closing (which I want to use to trigger automations). I hooked up a separate esphome-based sensor to trigger the automations instead. I can still use myQ to make the door open or close, but now get instant sensing of whether it's open or not.

hjdhjd commented 3 years ago

No worries! One thing I'd add...simply put: the resets are perfectly normal. They happen with HTTP connections all the time, and the myQ API can be flaky at times, though with this most recent move to better infrastructure, it seems to have improved a lot. My guess is that this is just debugging info that's inadvertently still being logged...it's not really that big a concern. The 401s are more the issue...that's problematic.

If you want to improve the granularity of updates here or in HA, I'm afraid I can't help you with that, but there may be config options to do so. For instance, in my homebridge plugin, update frequency is configurable if you really want to muck with it, though I would (strongly) recommend most people not mess with it unless they absolutely need to.

garyak commented 3 years ago

Up to 3.0.0b4. The custom_components.myq errors have returned:

2021-01-31 14:54:51 ERROR (MainThread) [custom_components.myq] Error fetching myq devices data: Error trying to re-authenticate to myQ service: Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer
2021-01-31 16:25:38 ERROR (MainThread) [custom_components.myq] Error fetching myq devices data: Error requesting data from https://accounts.myq-cloud.com/api/v6.0/accounts: 401 - Unauthorized

Also, the connect/disconnects have returned. Timing appears to coincide with the errors. connect

Earlier errors no longer appear in the log.

dseven commented 3 years ago

3.0.0b4 seems OK here. No connection resets logged. It noticed that my garage door had opened after about 40 seconds, which seems about the same as before.

ds7771 commented 3 years ago

3.0.0b4 is less stable for me.

I am seeing disconnects now, which appear to coincide with connection reset errors.

Previously the connection reset errors under 3.0.0b3 did not impact connection status.

ehendrix23 commented 3 years ago

FYI, just did an actual beta release figuring might be better to keep everyone on "same" release.

This is release 3.0.0b4 and I just released it. In your manifest.json set the following requirements line: "requirements": ["git+https://github.com/arraylabs/pymyq@3.0.0b4#pymyq==3.0.0b4", "pkce==1.0.2"],

If you already did a 3.0.0b4 then easiest would be to 1st just update current (do for example ==3.0.0b1), restart HASS, and once restarted update manifest.json.

From now on I'll be releasing new beta releases for folks to test here once I confirmed coding is working on my end.

Last changes are to hopefully reduce number of connect reset errors for authentication.

ehendrix23 commented 3 years ago

3.0.0b4 is less stable for me.

I am seeing disconnects now, which appear to coincide with connection reset errors.

Previously the connection reset errors under 3.0.0b3 did not impact connection status.

See post I just did to ensure you run latest.

ehendrix23 commented 3 years ago

Up to 3.0.0b4. The custom_components.myq errors have returned:

2021-01-31 14:54:51 ERROR (MainThread) [custom_components.myq] Error fetching myq devices data: Error trying to re-authenticate to myQ service: Error requesting data from https://partner-identity.myq-cloud.com/connect/authorize: [Errno 104] Connection reset by peer
2021-01-31 16:25:38 ERROR (MainThread) [custom_components.myq] Error fetching myq devices data: Error requesting data from https://accounts.myq-cloud.com/api/v6.0/accounts: 401 - Unauthorized

Introduced bug that I just discovered after leaving it run for a bit on my end. See my other post where I talk about actual releases. That will ensure what you test is something that has 1st gone through my own testing first and not anything that I'm still testing on. :-)

ds7771 commented 3 years ago

So far, looks good. Logs are quiet and disconnects have ceased.

GaryOkie commented 3 years ago

I concur - beta4 update is looking really good. Been running 10 hours straight and not a single warning/error or disconnect occurred.

Out of curiosity, what is the pymyq polling interval?

ehendrix23 commented 3 years ago

@ehendrix23 Take a look at the latest homebridge-myq - got lamps fully working with the new API. My educated guess was pretty close to begin with. 😄 Should’ve just trusted my instincts instead of thinking they’d keep more of the legacy commands. 😉

Looks like only thing was the commands to turn lamp on off from turnoff/turnoff to on/off but everything else was good. That is an impressive educated guess. :-)

ehendrix23 commented 3 years ago

I concur - beta4 update is looking really good. Been running 10 hours straight and not a single warning/error or disconnect occurred.

Out of curiosity, what is the pymyq polling interval?

Right now within pymyq we limit the polling interval to 20 seconds. So even if update is called more regularly then 20 seconds, pymyq will prevent that. HASS currently however polls every 60 seconds, within the updated myq component I've got it set to 30 seconds instead. Update interval from HASS is defined within const.py of the myq component using constant UPDATE_INTERVAL.

hjdhjd commented 3 years ago

I concur - beta4 update is looking really good. Been running 10 hours straight and not a single warning/error or disconnect occurred. Out of curiosity, what is the pymyq polling interval?

Right now within pymyq we limit the polling interval to 20 seconds. So even if update is called more regularly then 20 seconds, pymyq will prevent that. HASS currently however polls every 60 seconds, within the updated myq component I've got it set to 30 seconds instead. Update interval from HASS is defined within const.py of the myq component using constant UPDATE_INTERVAL.

If I may suggest a nuance to polling:

homebridge-myq polls every 12 seconds, by default. I’d suggest 12-15 seconds as a reasonable default. It also increases the polling resolution to every 3-5 seconds when it detects a state change. The argument being that as state changes are occurring, you should poll more to give people that “realtime” feeling. :smile: After a defined interval say 120-180 seconds (it’s actually longer in homebridge-myq, but I’ve been thinking of changing it in the near future) of no state changes, it returns back to it’s default polling interval.

You might want to consider something along these lines...it provides most of the benefits of both worlds, in practice, given that most folks aren’t constantly opening and closing their garage doors. :smile:

hjdhjd commented 3 years ago

@ehendrix23 Take a look at the latest homebridge-myq - got lamps fully working with the new API. My educated guess was pretty close to begin with. 😄 Should’ve just trusted my instincts instead of thinking they’d keep more of the legacy commands. 😉

Looks like only thing was the commands to turn lamp on off from turnoff/turnoff to on/off but everything else was good. That is an impressive educated guess. :-)

Right?!!! :smile:

If I was that good at taking an educated guess to the lottery...I’d probably be able to retire being that close!

brettonw commented 3 years ago

homebridge-myq polls every 12 seconds, by default. I’d suggest 12-15 seconds as a reasonable default. It also increases the polling resolution to every 3-5 seconds when it detects a state change. The argument being that as state changes are occurring, you should poll more to give people that “realtime” feeling. 😄 After a defined interval say 120-180 seconds (it’s actually longer in homebridge-myq, but I’ve been thinking of changing it)of no state changes, it returns back to it’s default polling interval.

UPVOTE. This is a spectacular idea, as the long HA polling interval gives a sense of unresponsiveness when you need to open and then close the door in short succession.

arraylabs / pymyq

MyQ API Update. #67