search

twrecked / pyaarlo

Asynchronous Arlo Component for Python
GNU Lesser General Public License v3.0
50 stars 30 forks source link

authentication error #121

Open vzlgdu opened 1 year ago

vzlgdu commented 1 year ago

Hi, starting from this morning I'm receiving authentication errore in my python scritp. It's running with no problem starting from many months ago...

What's happening?

Thanks

Guido

stefanludwig commented 1 year ago

Hi, just caught up with this thread. I'm seeing the same issue body-error=JSONDecodeError Using this:

arlo = pyaarlo.PyArlo( username=USERNAME, password=PASSWORD,
                       tfa_type='email', tfa_source='imap', tfa_host='<redacted>')

and as far as I see above, there is no resolution yet. Correct?

twrecked commented 1 year ago

I'll try and get something out later but two things are happening to me:

stefanludwig commented 1 year ago

As a sidenote, I think Arlo could do a much better job at enabling programmatic interfaces. I wish I had known this (i.e. researched it) before I bought 10 cameras for two houses…

shissam commented 1 year ago

I'll try and get something out later but two things are happening to me:

  • I need the updated cache header settings
  • cloudscraper is not playing well with the sse channel

@twrecked a bit of feedback... I "shoe-horned" pyaarlo 0.8.0b9 into the hass-aarlo 0.8.0a14 HA's custom-components folder (as pip3 could not yet resolve pyaarlo>=0.8.0b9) and restarted HA - and I am "back in baby"!

what is strange to me is that trying my (now) legacy pyaarlo custom code with pyaarlo 0.8.0b9 I am still not able to get past cloudfare guard and I am sure I destroyed all the pickle files and furthermore I could see the header changes (no cache, etc.) in the debug.

But for now I am thrilled that HA is back up--BTW: just before trying pyaarlo 0.8.0b9, I confirmed that pyaarlo 0.8.0b8 was still failing at the cloudfare perimeter.

m0urs commented 1 year ago

I still had the same issue with Cloudflare after updating to 0.8.0b9. However, I think I have solved it :-)

I needed to add the following parameter to the Cloudscraper call:

self._session = cloudscraper.create_scraper(ecdhCurve='secp384r1')

In addition I am using the following custom User agent string 8not sure if really necessary):

user_agent='!Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58'

After I did that, I was able to login again :-)

m0urs commented 1 year ago

Ok, although I am now able to login, setting the mode for my cameras are no longer working. I am using the SSE backend. I tried to switch to MQTT but in this case there is the following error after login:

2023-07-05 09:46:52,023 - pyaarlo - DEBUG - backend: request-end=200
2023-07-05 09:46:52,023 - pyaarlo - DEBUG - multilocation is False
2023-07-05 09:46:52,024 - pyaarlo - DEBUG - backend: (re)starting mqtt event loop
2023-07-05 09:46:52,024 - pyaarlo - DEBUG - backend: mqtt: client_id=user_38D2FSKA-183-52572877_6817086917
2023-07-05 09:46:52,026 - pyaarlo - DEBUG - backend: mqtt: host=mqtt-cluster.arloxcld.com, check=True, transport=tcp
2023-07-05 09:46:52,028 - pyaarlo - ERROR - general-error=gaierror
Traceback (most recent call last):
  File "/opt/fhem/arlo-fhem/pyaarlo/backend.py", line 526, in _mqtt_main
    self._event_client.connect(self._arlo.cfg.mqtt_host, port=443, keepalive=60)
  File "/opt/fhem/.local/lib/python3.8/site-packages/paho/mqtt/client.py", line 914, in connect
    return self.reconnect()
  File "/opt/fhem/.local/lib/python3.8/site-packages/paho/mqtt/client.py", line 1044, in reconnect
    sock = self._create_socket_connection()
  File "/opt/fhem/.local/lib/python3.8/site-packages/paho/mqtt/client.py", line 3685, in _create_socket_connection
    return socket.create_connection(addr, timeout=self._connect_timeout, source_address=source)
  File "/usr/local/lib/python3.8/socket.py", line 787, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

2023-07-05 09:46:52,029 - pyaarlo - DEBUG - backend: waiting for stream up

Do I miss something? Does it makes sense at all to switch to MQTT? Advantages/Disadvantages?

Or should we try to fix setting the mode in SSE mode?

Here is some debug information while trying to arm my cameras:

debug.log

twrecked commented 1 year ago

@m0urs I don't know what is happening with mqtt so leave it for sse for now. The devices still report supporting it but nothing seems to use it and they changed the host name to something I don't know.

m0urs commented 1 year ago

Ok, so in this case (SSE), could you please have a look at the debug.log as I am no longer able to change the status of the devices (e.g. to set the armed state etc.)

ThomasHammant commented 1 year ago

Ok, so in this case (SSE), could you please have a look at the debug.log as I am no longer able to change the status of the devices (e.g. to set the armed state etc.)

I'm seeing exactly the same. Couldn't connect at all until version 17 since just before the weekend. Now after upgrading and waiting overnight a reboot of HA resulted in connecting for the first time in a few days. However can't change any of the states e.g. to arm a camera.

Happy to share any logs etc. you need if this helps just let me know.

This is with the SSE setting by the way.

pfcurtis commented 1 year ago

0.8.0b9 + issue #124 resolved all my issues.

This is a stand alone (not HA) python 3.9 application. Tested both in a container and a venv with only the pyaarlo-0.8.0b9 as the requirement. Tested also with python 3.11 under the same conditions.

------------------ --------
certifi            2023.5.7
cffi               1.15.1
charset-normalizer 3.1.0
click              8.1.3
cloudscraper       1.2.71
cryptography       41.0.1
idna               3.4
paho-mqtt          1.6.1
pip                23.0.1
pyaarlo            0.8.0b9
pycparser          2.21
pycryptodome       3.18.0
pyparsing          3.1.0
requests           2.31.0
requests-toolbelt  1.0.0
setuptools         67.6.1
Unidecode          1.3.6
urllib3            2.0.3
shissam commented 1 year ago

I'll try and get something out later but two things are happening to me:

  • I need the updated cache header settings
  • cloudscraper is not playing well with the sse channel

@twrecked a bit of feedback... I "shoe-horned" pyaarlo 0.8.0b9 into the hass-aarlo 0.8.0a14 HA's custom-components folder (as pip3 could not yet resolve pyaarlo>=0.8.0b9) and restarted HA - and I am "back in baby"!

what is strange to me is that trying my (now) legacy pyaarlo custom code with pyaarlo 0.8.0b9 I am still not able to get past cloudfare guard and I am sure I destroyed all the pickle files and furthermore I could see the header changes (no cache, etc.) in the debug.

But for now I am thrilled that HA is back up--BTW: just before trying pyaarlo 0.8.0b9, I confirmed that pyaarlo 0.8.0b8 was still failing at the cloudfare perimeter.

@twrecked, @m0urs, @stefanludwig : For those of you NOT using HA and using pyaarlo directly in your own custom app--I discovered a potential cause, but I still need time to completely investigate.

Basically, I copied my pyaarlo custom app into the HA container, exec'd into the container and ran the custom app against the pyaarlo 0.8.0b9 and it worked. But the same app outside the container continues to fail withpyaarlo 0.8.0b9 as noted above. Some initial findings:

My OS has about 107 pypi packages installed (whereas HA has 1,207 installed) that I need to process through to see if it is a version out of date with cloudfare.

If anyone has an idea as to which pypi package might be the culprit - shout it out! If I discover the offender, I'll post back.

MTW: @m0urs I HAVE NOT added your hint re: self._session = cloudscraper.create_scraper(ecdhCurve='secp384r1') - I did not seem to need it under the HA container environment.

m0urs commented 1 year ago

Does adding the ecdhCurve parameter get you legacy app to work outside of the HA container!

shissam commented 1 year ago

Does adding the ecdhCurve parameter get you legacy app to work outside of the HA container!

I'll give it a try here in an hour or two - and report back, my suspicion is unlikely - but I have been surprised way too many times to rule it out. The cloudscraper README.md reports


--
# Some servers require the use of a more complex ecdh curve than the default "prime256v1"
# It may can solve handshake failure
scraper = cloudscraper.create_scraper(ecdhCurve='secp384r1')
```python

But there did not appear to be any code in `cloudscraper` itself actually employing that tactics on its own.
shissam commented 1 year ago

Does adding the ecdhCurve parameter get you legacy app to work outside of the HA container!

I'll give it a try here in an hour or two - and report back, my suspicion is unlikely - but I have been surprised way too many times to rule it out. The cloudscraper README.md reports

--
# Some servers require the use of a more complex ecdh curve than the default "prime256v1"
# It may can solve handshake failure
scraper = cloudscraper.create_scraper(ecdhCurve='secp384r1')
```python

But there did not appear to be any code in `cloudscraper` itself actually employing that tactics on its own.

@m0urs, uuuuugggggghhhhh!, no it made no difference:

$ python3 arlo-mon2FA.py 
init login config
reading login config
grabing secrets
reading /home/pi/.ssh/arlo-mon2FA.secrets
logging in
2023-07-05 13:49:42,199 - pyaarlo - INFO - pyarlo 0.8.0b9 starting...
2023-07-05 13:49:42,331 - pyaarlo - WARNING - body-error=JSONDecodeError
2023-07-05 13:49:43,355 - pyaarlo - WARNING - body-error=JSONDecodeError
2023-07-05 13:49:44,387 - pyaarlo - WARNING - body-error=JSONDecodeError
2023-07-05 13:49:45,389 - pyaarlo - ERROR - authentication failed
Error: 'PyArlo' object has no attribute '_bases'
monitoring bases
not successfully connected
$ grep cloudscraper.create_scraper ../pyaarlo/pyaarlo/backend.py 
            self._session = cloudscraper.create_scraper(ecdhCurve='secp384r1')
$ ls -ltr ../pyaarlo/pyaarlo/__pycache__/backend.cpython-37.pyc 
-rw-r--r-- 1 pi pi 27351 Jul  5 13:49 ../pyaarlo/pyaarlo/__pycache__/backend.cpython-37.pyc
twrecked commented 1 year ago

@shissam I also found out the delay option in cloudscraper doesn't do anything...

But I think you are onto something with the versioning, my pyaarlo works for me and these are my versions.

python is version 3.10 cloudscraper=1.2.71 urllib3=2.0.3 requests=2.31.0 ** requests-toolbelt=1.0.0

This is Ubuntu 22..04. I have some PIs somewhere around here, I'll try one of them later.

shissam commented 1 year ago

@shissam I also found out the delay option in cloudscraper doesn't do anything...

But I think you are onto something with the versioning, my pyaarlo works for me and these are my versions.

python is version 3.10 cloudscraper=1.2.71 urllib3=2.0.3 requests=2.31.0 ** requests-toolbelt=1.0.0

This is Ubuntu 22..04. I have some PIs somewhere around here, I'll try one of them later.

@twrecked I got lazy. I created a python venv under/on my host raspbian OS using Python 3.7.3.

I simply (re-)cloned pyaarlo 0.8.0b9 in the venv working folder, upgraded pip, and ran my python pyaarlo app while using pip install repeatedly to resolve packages and specifying the same version of the package as stated in the HA container env.

It worked. I believe these steps can reproduce the env:

Commands used to set up the venv:

$ mkdir pyaarlo-dbg
$ python3 -m venv pyaarlo-dbg/asis-venv
$ source pyaarlo-dbg/asis-venv/bin/activate
$ ~/pyaarlo-dbg/asis-venv/bin/python3 -m pip install --upgrade pip
$ ~/pyaarlo-dbg/asis-venv/bin/python3 -m pip install 'cloudscraper==1.2.71'
$ ~/pyaarlo-dbg/asis-venv/bin/python3 -m pip install 'paho-mqtt==1.6.1'
$ ~/pyaarlo-dbg/asis-venv/bin/python3 -m pip install unidecode
$ ~/pyaarlo-dbg/asis-venv/bin/python3 -m pip install 'cryptography==40.0.2'

Here are the packages installed in the venv and the corresponding version from the Host OS:

                   Venv       Host (raspbian)
Package            Version    Version
------------------ --------   -------
certifi            2023.5.7   2018.8.24
cffi               1.15.1     1.15.0
charset-normalizer 3.1.0      (not listed by pip3)
cloudscraper       1.2.71     1.2.71
cryptography       40.0.2     2.6.1
idna               3.4        2.6
paho-mqtt          1.6.1      1.6.1
pip                23.1.2     18.1
pkg_resources      0.0.0      (not listed by pip3)
pycparser          2.21       2.21
pyparsing          3.1.0      3.0.7
requests           2.31.0     2.21.0
requests-toolbelt  1.0.0      0.9.1
setuptools         40.8.0     40.8.0
Unidecode          1.3.6      1.3.2
urllib3            2.0.3      1.24

One or some combinations of these may be the culprit. That is unless this mixture is sensitive to python 3.7.3 which I am still using on the Host OS

twrecked commented 1 year ago

Thanks for trying that. cloudscraper relies on requests. And requests seems to use urllib3. So that's 2 to try. But I could also see cryptography causing issues.

I could add some versions to the requirements.txt file, that might help, right now there are non specified.

shissam commented 1 year ago

Thanks for trying that. cloudscraper relies on requests. And requests seems to use urllib3. So that's 2 to try. But I could also see cryptography causing issues.

I could add some versions to the requirements.txt file, that might help, right now there are non specified.

i am by far "not" a python expert (nor do I play one on TV) - but there is one thing that is nagging me - say, if I upgrade from cloudscraper 1.2.58 to 1.2.71 (for example) would a pip3 upgrade, upgrade all the sub-dependencies? I could swear in recent weeks I upgraded cloudscraper from 1.2.58 to 1.2.71 and in that process, those old version number were there and are still there (as shown in that table).

i'll see about upgrading those sub-dependencies one by one and see which one gets me back - I'll have to get to that in the next day or so and i'll report back if no one else does.

m0urs commented 1 year ago

Ok, as I had the same versions as @twrecked I just upgraded all to the most current version, just to give it a try. That is what I now have installed:

Python 3.10.12

Package            Version
------------------ --------
asn1crypto         1.5.1
certifi            2023.5.7
cffi               1.15.1
chardet            5.1.0
charset-normalizer 3.1.0
click              8.1.3
cloudscraper       1.2.71
cryptography       41.0.1
Cython             0.29.36
gevent             22.10.2
greenlet           2.0.2
idna               3.4
mock               5.0.2
paho-mqtt          1.6.1
pip                23.1.2
pycparser          2.21
pycryptodome       3.18.0
pyparsing          3.0.9
pysam              0.21.0
python-dateutil    2.8.2
requests           2.31.0
requests-toolbelt  1.0.0
setuptools         68.0.0
six                1.16.0
Unidecode          1.3.6
urllib3            2.0.3
websocket          0.2.1
wheel              0.40.0
zipp               3.15.0
zope.event         5.0
zope.interface     6.0

However @shissam:

I CANNOT connect if I do NOT add the parameter "ecdhCurve='secp384r1'" to the clouscraper call. I tried it several times. Without that parameter, the login fails with the known Cloudflare message, with that parameter the login works without a problem ...

But unfortunately, I still cannot set my modes for the cameras anymore :-( So hopefully @twrecked you would be able to find out what is going wrong here? I think we had a similar issue some time ago already.

shissam commented 1 year ago

Ok, as I had the same versions as @twrecked I just upgraded all to the most current version, just to give it a try. That is what I now have installed:

Python 3.10.12

Package            Version
------------------ --------
asn1crypto         1.5.1
certifi            2023.5.7
cffi               1.15.1
chardet            5.1.0
charset-normalizer 3.1.0
click              8.1.3
cloudscraper       1.2.71
cryptography       41.0.1
Cython             0.29.36
gevent             22.10.2
greenlet           2.0.2
idna               3.4
mock               5.0.2
paho-mqtt          1.6.1
pip                23.1.2
pycparser          2.21
pycryptodome       3.18.0
pyparsing          3.0.9
pysam              0.21.0
python-dateutil    2.8.2
requests           2.31.0
requests-toolbelt  1.0.0
setuptools         68.0.0
six                1.16.0
Unidecode          1.3.6
urllib3            2.0.3
websocket          0.2.1
wheel              0.40.0
zipp               3.15.0
zope.event         5.0
zope.interface     6.0

However @shissam:

I CANNOT connect if I do NOT add the parameter "ecdhCurve='secp384r1'" to the clouscraper call. I tried it several times. Without that parameter, the login fails with the known Cloudflare message, with that parameter the login works without a problem ...

But unfortunately, I still cannot set my modes for the cameras anymore :-( So hopefully @twrecked you would be able to find out what is going wrong here? I think we had a similar issue some time ago already.

@m0urs - I too am unable to set base mode - this error is mentioned above in this issue report as well as over on the HA issue thread at https://github.com/twrecked/hass-aarlo/issues/767 this is what I am seeing when I set base.mode = 'disarmed'

base: name=(redacted),state=available,mode=Armed Day Home,sched=True,schednam=schedule.1
['disarmed', 'armed', 'armed day home', 'armed home with driveway alert', 'armed night home', 'schedule.1']
disarming...
2023-07-05 16:03:41,217 - pyaarlo - WARNING - attempt 1: error in response when setting mode=
{'data': {'error': 'AUTO-5050',
          'message': 'Failed to update Automation Definitions',
          'reason': 'Unable to upload active automation. Please try again'},
 'success': False}
m0urs commented 1 year ago

YEs, that is the same error I see here. I uploaded a debug log under https://github.com/twrecked/pyaarlo/issues/121#issuecomment-1621234020 So it seems that is a bigger problem :-(

twrecked commented 1 year ago

I'm seeing that as well. I'll take a look.

twrecked commented 1 year ago

I'll push a fix later, I have to head into work, but for now, un-comment the "SchemaVersion": "1" line.

I took it out to make sure the headers matched for the sse testing and forgot to add it back.

herveaurel commented 1 year ago

I'll push a fix later, I have to head into work, but for now, un-comment the "SchemaVersion": "1" line.

I took it out to make sure the headers matched for the sse testing and forgot to add it back.

Yes THKS !!!! ✌🏻🎉

m0urs commented 1 year ago

I'll push a fix later, I have to head into work, but for now, un-comment the "SchemaVersion": "1"

Thank you so much! Works again :-)

shissam commented 1 year ago

I'll push a fix later, I have to head into work, but for now, un-comment the "SchemaVersion": "1" line.

I took it out to make sure the headers matched for the sse testing and forgot to add it back.

yep - did the complete git pull and my redeploy ... all appears good and mode setting is back!

@twrecked thank you for all you do!

bjia56 commented 1 year ago

@twrecked fwiw, on the Scrypted Arlo plugin side, I am working on a Cloudflare fix using this package: https://pypi.org/project/curl-cffi

From my personal testing, Cloudflare is picky about the initial login connection, but once a session token has been returned, further API calls are less sensitive and seem to go through ok. I'm doing initial testing with using curl-cffi only for the initial auth request, then switching to cloudscraper for future API and SSE connections using the token returned from the auth. Preliminary results are promising.

Unfortunately there are no 32 bit raspberry pi builds of curl-cffi, so not sure if that's a concern on the HA side.

twrecked commented 1 year ago

@bjia56 Thanks, I'll definitely take a look at that package. Might be nice to offer it as an option.

And pyaarlo already uses a non cloudscraper socket if it detects a valid cached authentication token (which are good for 14 days) and for the sse connection. In fact, sse doesn't seem compatible with cloudscraper, it just hangs.

It easy to modify pyaalo to ditch the cloudscraper after authentication as well, so I might try that.

shissam commented 11 months ago

@twrecked as noted above, pyaarlo "0.8.0b10 worked (and was working since July 6 until this morning Jul 18) -- I did notice that hass-arlo's issue https://github.com/twrecked/hass-aarlo/issues/778 were also noting issues requiring @m0urs detected change "ecdhCurve='secp384r1'" and removing "Source": "arloCamWeb", from the header in backend.py

I made those two changes this morning and I am back in again. I can only suspect that changes on cloudflare may not necessarily roll out to all their zones/servers at the same time and I may have been "benefiting" from older servers. After seeing the hass-arlo reports, I suspected it was only a matter of time before I had to make those changes.

diff --git a/pyaarlo/backend.py b/pyaarlo/backend.py
index 9561f7b..221ef03 100644
--- a/pyaarlo/backend.py
+++ b/pyaarlo/backend.py
@@ -688,7 +688,7 @@ class ArloBackEnd(object):
             # "Sec-Fetch-Dest": "empty",
             # "Sec-Fetch-Mode": "cors",
             # "Sec-Fetch-Site": "same-site",
-            "Source": "arloCamWeb",
+            # "Source": "arloCamWeb",
             "User-Agent": self._user_agent,
             "X-User-Device-Automation-name": "QlJPV1NFUg==",
             "X-User-Device-Id": self._user_device_id,
@@ -896,7 +896,8 @@ class ArloBackEnd(object):
         # If token looks invalid we'll try the whole process.
         get_new_session = days_until(self._expires_in) < 2
         if get_new_session:
-            self._session = cloudscraper.create_scraper()
+            # self._session = cloudscraper.create_scraper()
+            self._session = cloudscraper.create_scraper(ecdhCurve='secp384r1')
             self.debug("oldish session, getting a new one")
             if not self._auth():
                 return False

I should’ve re-tried without removing "Source": "arloCamWeb”,— but I am using everyone else’s success.

twrecked commented 11 months ago

And I've just update and checked in that success :)

Can somebody double check and make sure the master branch is working ok?

shissam commented 11 months ago

@twrecked recent pull for pyarlo 0.8.0b11 is working as I expected for these changes. Thank you!

tdeckers commented 11 months ago

Confirmed.. that latest version iss now working fine. Thanks!

vinnygambiny commented 10 months ago

I try the 0.8.0b11 and it's not working for me 😞

shissam commented 10 months ago

@twrecked recent pull for pyarlo 0.8.0b11 is working as I expected for these changes. Thank you!

@twrecked so, it happened again JSON decode errors. I even tried a most recent pull of pyaarlo this morning (commit 4a0763fc...) still got pyaarlo - WARNING - body-error=JSONDecodeError (really no surprise there given the last change).

the fix for me today was that of @Molnigt over at hass-arlo #778 where cloudscraper.create_scraper() was modified to remove all customizations (i.e., no args for ecdhCurve).

call this a silly hypothesis, but I am continuing to suspect that cloudflare continues to run different productions in places.

have fun!

meavydev commented 10 months ago

It just failed for me. It worked 1.5 hours ago and now it isn't (latest b11 build). With debugging on I see access token revoked and then the JSONDecodeError.

twrecked commented 10 months ago

@shissam I agree with your hypothesis, and it's part of the reason I'm loathed to keep changing this code about. Be we do seem to have a few simple things to try:

If all that fails wait before trying again.

I'll also take my monthly look at the headers in Chrome to see if the web requests have changed.

meavydev commented 8 months ago

It is working again for me, so thanks.

meavydev commented 8 months ago

Blimey... short lived working. Back to JSON Decode error with 0.8.0.A14.

meavydev commented 8 months ago

Ah my fault. I'm not a Python expert and hadn't realised that pip was updating one location and then the webapp was using a different one. Now both are set to 0.8.0.1 and it seems to be working again, so hopefully it will stay working this time...