403 Client Error: Forbidden for url: https://ocapi-app.arlo.com/api/auth

[scuc]

Is anyone else seeing this 403 Client Error? I started to see it on May 09, my script runs automated so I didn't actually notice this issue in my logs until last week.
I thought maybe it was an MFA issue, so I've tried to add MFA as explained in the wiki, but I cant even get past the 403 error.

What version of Python are you using (python -V)?

Python 3.10.4

What operating system and processor architecture are you using (python -c 'import platform; print(platform.uname());')?

('Darwin', 'MacPro-Mojave.local', '18.7.0', 'Darwin Kernel Version 18.7.0: Tue Jun 22 19:37:08 PDT 2021; root:xnu-4903.278.70~1/RELEASE_X86_64', 'x86_64', 'i386')

Which Python packages do you have installed (run the pip freeze or pip3 freeze command and paste output)?

arlo==1.2.59 (also test  with .64 version)
cachetools==5.3.1
certifi==2023.5.7
cffi==1.15.0
charset-normalizer==3.1.0
click==8.1.3
cloudscraper==1.2.60
cryptography==37.0.2
google-api-core==2.11.1
google-api-python-client==2.90.0
google-auth==2.21.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==1.0.0
googleapis-common-protos==1.59.1
httplib2==0.22.0
idna==3.4
monotonic==1.6
oauthlib==3.2.2
paho-mqtt==1.6.1
pickle-mixin==1.0.2
protobuf==4.23.3
pyaarlo @ git+https://github.com/twrecked/pyaarlo@2d6941dc903fe2fca01ac9d82fe708d1299ebe81
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
pycryptodome==3.14.1
pyparsing==3.1.0
PySocks==1.7.1
PyYAML==6.0
requests==2.31.0
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
rsa==4.9
six==1.16.0
sseclient==0.0.22
Unidecode==1.3.4
uritemplate==4.1.1
urllib3==1.24

Which version of ffmpeg are you using (ffmpeg -version)?

ffmpeg version 1.2-tessus Copyright (c) 2000-2013 the FFmpeg developers
  built on Mar 15 2013 01:18:55 with llvm-gcc 4.2.1 (LLVM build 2336.1.00)
  configuration: --prefix=/Users/tessus/data/ext/ffmpeg/sw --as=yasm --extra-version=tessus --disable-shared --enable-static --disable-ffplay --enable-gpl --enable-pthreads --enable-postproc --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid --enable-libspeex --enable-bzlib --enable-zlib --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libxavs --enable-version3 --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvpx --enable-libgsm --enable-libopus --enable-fontconfig --enable-libfreetype --enable-libass --enable-filters --enable-runtime-cpudetect
  libavutil      52. 18.100 / 52. 18.100
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.104 / 54. 63.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 42.103 /  3. 42.103
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100

Which Arlo hardware do you have (camera types - [Arlo, Pro, Q, etc.], basestation model, etc.)?

not relevant

What did you do?

This code was working for me up without MFA until May 09 2023 - I thought maybe it was an MFA issue, so I added MFA to the code, but still getting the same errrors.

from arlo import Arlo
from datetime import timedelta, date, datetime
import logging
# from pathlib import Path

import config
import check_subdirs as chksub

config = config.get_config()

USERNAME = config['creds']['username']
PASSWORD = config['creds']['password']
MFA = config['creds']['mfa']

logger = logging.getLogger(__name__)

ROOTPATH = config['paths']['root_path']

def download_mp4s(): 
    """
    Iterate over the videos in the arlo S3 bucket, compare against videos stored locally, dowload 
    all new files, rename downloaded files with a datetime, skip those that have already been downloaded. 
    """
    # Instantiating the Arlo object automatically calls Login(), which returns an oAuth token that gets cached.
    # Subsequent successful calls to login will update the oAuth token.
    try: 
        arlo = Arlo(USERNAME, PASSWORD, MFA)
        logger.info("Successfully logged into Arlo account")

        today = (date.today()-timedelta(days=0)).strftime("%Y%m%d")
        start_date = (date.today()-timedelta(days=30)).strftime("%Y%m%d")

        # Get all of the recordings for a date range.
        library = arlo.GetLibrary(start_date, today)
        logger.info("Connected to the Arlo Library")
        devices = arlo.GetDevices()
        # print(devices)

        device_list = []

        for device in devices:
            deviceId = device["deviceId"]
            deviceName = device["deviceName"]
            device_dict = {device["deviceId"]: device["deviceName"]}
            device_list.append(device_dict)

        logger.info(f"Devices found in library: {device_list}")

        # Iterate through the recordings in the library.
        download_count = 0
        for recording in library:

            deviceId = recording["deviceId"]

            deviceNum = next(i for i,d in enumerate(device_list) if deviceId in d)

            deviceName = device_list[deviceNum][deviceId]

            videofilename = datetime.fromtimestamp(int(recording['name'])//1000).strftime('%Y-%m-%d_%H-%M-%S') + '_' + recording['uniqueId'] + '.mp4'

            ###################
            # The videos produced by Arlo are pretty small, even in their longest, best quality settings,
            # but you should probably prefer the chunked stream (see below).
            ##################
            #    # Download the whole video into memory as a single chunk.
            #    video = arlo.GetRecording(recording['presignedContentUrl'])
            #    with open('videos/'+videofilename, 'wb') as f:
            #        f.write(video)
            #        f.close()
            ################33

            root_dir = (ROOTPATH + deviceName + '/')
            mp4_path = chksub.check_subdirs(root_dir, deviceName, videofilename)

            if mp4_path.is_file() is not True:
                stream = arlo.StreamRecording(recording['presignedContentUrl'])
                with open(str(mp4_path), 'wb') as f:
                    for chunk in stream: 
                        f.write(chunk)
                    f.close()
                    download_msg = 'Downloaded '+ videofilename+' from '+ recording['createdDate']+'.'
                    dowload_path_msg = "Download path: " + str(mp4_path)
                    download_count += 1
                    logger.info(download_msg)
                    logger.info(dowload_path_msg)
            else:
                # Get video as a chunked stream; this function returns a generator.
                skip_msg = "Skipping: " + str(videofilename)
                logger.debug(skip_msg)
                continue

        # Delete all of the videos you just downloaded from the Arlo library.
        # Notice that you can pass the "library" object we got back from the GetLibrary() call.
        # result = arlo.BatchDeleteRecordings(library)

        if download_count != 0: 
            logger.info(f"Total videos downloaded: {download_count}")
        else: 
            logger.info(f"No new videos for download.")
        return

    except Exception as e:
        logger.exception(e)

if __name__ == '__main__':
    download_mp4s()

What did you expect to see?

I use the code listed above to download a local copy of my arlo videos.

What did you see instead?

    ================================================================
                ARLO Download Script - Start
                    Wednesday, 28. June 2023 09:46PM
    ================================================================

2023-06-28 21:46:34,815 | ERROR | Function: download_mp4s() | Line 103 | 403 Client Error: Forbidden for url: https://ocapi-app.arlo.com/api/auth
Traceback (most recent call last):
  File "/Users/myname/_Github/ARLO-S3-Download/arlo_download.py", line 27, in download_mp4s
    arlo = Arlo(USERNAME, PASSWORD, MFA)
  File "/Users/myname/_Github/ARLO-S3-Download/.venv/lib/python3.10/site-packages/arlo.py", line 69, in __init__
    self.LoginMFA(username, password, google_credential_file)
  File "/Users/myname/_Github/ARLO-S3-Download/.venv/lib/python3.10/site-packages/arlo.py", line 201, in LoginMFA
    auth_body = self.request.post(
  File "/Users/myname/_Github/ARLO-S3-Download/.venv/lib/python3.10/site-packages/request.py", line 80, in post
    return self._request(url, 'POST', params=params, headers=headers, raw=raw)
  File "/Users/myname/_Github/ARLO-S3-Download/.venv/lib/python3.10/site-packages/request.py", line 56, in _request
    r.raise_for_status()
  File "/Users/myname/_Github/ARLO-S3-Download/.venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://ocapi-app.arlo.com/api/auth
2023-06-28 21:46:34,820 | INFO | Function: main() | Line 89 | 
        ================================================================
                    ARLO Download - Complete
                        Wednesday, 28. June 2023 09:46PM
        ================================================================

Does this issue reproduce with the latest release?

yes. same results. in .59 and .64, ive tried connecting through a VPN to see if I my IP was being blocked, i can also confirm that I have tried multiple user accounts - in each case I can log in just fine through the arlo web portal, but get the 403 when using the Arlo code.

[scuc]

FYI, it looks like scrypted issued a patch to deal with the exact same type of 403 issue. Also, the issue appeared around May 10 right around same time as the issue that I am seeing.

https://github.com/koush/scrypted/pull/809/commits/0a020f21613a52e9621757a1689706d522af770b

Trying to understand exactly what they did to fix the issue, no comments in the code.

[misterek]

Just took a quick look, but it looks to me like the assumption is that CloudFlare is causing the issue. So they hardcoded some IP's direct to Arlo (in base64), and if the main one fails, they try to find one of those with the correct SSL cert, and use that instead.

Doesn't seem like a great option, since those IP's can change at any time (and if I checked right, at lest one is already not pointing at Arlo.)

[tomelsj]

Hmm, not even sure the scrypted solution works at all anymore. The API is protected behind CloudFront. Have not found any way to access it. And all the IP:s are broken as well.

Perhaps better luck building a "private server" for the Arlo cameras.

[misterek]

I'd guess CloudFlare is doing some sort of TLS fingerprinting or similar. I played around a bit, both using Cloudscraper and swapping around the TLS cipher order. Didn't have any luck.

I'll wait a bit to see what anyone else comes up with, but at this point I may just go with another vendor if Arlo doesn't want me to have access to my own recordings.

[misterek]

Well, given this: https://github.com/koush/scrypted/commit/8eb533c22025f0e056be885b07450723f0fbc884 Maybe they've found a way around using cloud scraper. Note that they had to update their IP list as well. I expect that'll have to be done regularly.

[m0urs]

I had a similar issue with Arlo and Cloudscraper based on another Python module (https://github.com/m0urs/arlo-fhem). For me, the Cloudflare login issue has been resolved by adding this parameter to the cloudscraper call:

self._session = cloudscraper.create_scraper(ecdhCurve='secp384r1')

together with Cloudscraper 1.2.71 and the following User Agent String:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58

Not sure if the last two things are related to that, but the Cloudscraper parameter definitely has been necessary for me. Maybe it could resolve the issue for others as well ...

[scuc]

@m0urs thank you for the suggestions, but I'm unclear as to where exactly i should add the cloud scraper call into my code. A search of the arlo library has no ref to cloudscraper.

[m0urs]

To be honest, I do not know this code here, but I guess that something similar is used to handle the Cloudflare checking for HTTP requests. Maybe @jeffreydwalter should know how this code is doing it and maybe there are similar settings.

[tomelsj]

In arlo.py at line 139, something like this could be done: self.request = cloudscraper.create_scraper(ecdhCurve='secp384r1') Instead of Request() Need to import cloudscraper too like import cloudscraper along with other imports. I cannot verify this actually works at this moment.

[scuc]

@tomelsj thanks for the suggestions, i tried it, unfortunately, still get a 403 error, so frustrating - requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://ocapi-app.arlo.com/api/auth

[m0urs]

Did you wait a day between the last login try without the parameter and the next try with the parameter? If not, please wait some hours until Cloudflare has unbanned your IP address.

[scuc]

@m0urs yes, i updated my code and waited a few days before trying again unfortunately I still see a 403-Forbidden response.

[scuc]

I still have not been able to get this to work. is there someone who actually has the login working with Arlo library? if so, would you be able to send a PR for this package to addresses this issue?

[Joreid]

Any updates for this error?

[ahass-thedev]

Hoping for any leads/fixes for this error. Unsure if it is possible to proceed without a fix

[scuc]

@bjia56 - im circling back to this issue - trying out your fix, but getting the error below. not sure what's wrong here - have my gmail credential expired?

2023-10-13 13:21:04,566 | INFO | Function: autodetect() | Line 49 | file_cache is only supported with oauth2client<4.0.0
2023-10-13 13:21:09,778 | ERROR | Function: download_mp4s() | Line 103 | ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})
Traceback (most recent call last):
  File "/Users/username/Documents/Github/arlo-cloudflarefix/arlo_download.py", line 27, in download_mp4s
    arlo = Arlo(USERNAME, PASSWORD, google_credential_file=MFA)
  File "/Users/username/Documents/Github/arlo-cloudflarefix/arlo.py", line 69, in __init__
    self.LoginMFA(username, password, google_credential_file)
  File "/Users/username/Documents/Github/arlo-cloudflarefix/arlo.py", line 256, in LoginMFA
    ).execute()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/http.py", line 923, in execute
    resp, content = _retry_request(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/http.py", line 191, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google_auth_httplib2.py", line 209, in request
    self.credentials.before_request(self._request, method, uri, request_headers)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/auth/credentials.py", line 134, in before_request
    self.refresh(request)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/oauth2/credentials.py", line 319, in refresh
    ) = reauth.refresh_grant(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/oauth2/reauth.py", line 349, in refresh_grant
    _client._handle_error_response(response_data, retryable_error)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/oauth2/_client.py", line 69, in _handle_error_response
    raise exceptions.RefreshError(
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})
2023-10-13 13:21:09,783 | INFO | Function: main() | Line 89 | 

[scuc]

@bjia56 - im circling back to this issue - trying out your fix, but getting the error below. not sure what's wrong here - have my gmail credential expired?

2023-10-13 13:21:04,566 | INFO | Function: autodetect() | Line 49 | file_cache is only supported with oauth2client<4.0.0
2023-10-13 13:21:09,778 | ERROR | Function: download_mp4s() | Line 103 | ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})
Traceback (most recent call last):
 File "/Users/username/Documents/Github/arlo-cloudflarefix/arlo_download.py", line 27, in download_mp4s
   arlo = Arlo(USERNAME, PASSWORD, google_credential_file=MFA)
 File "/Users/username/Documents/Github/arlo-cloudflarefix/arlo.py", line 69, in __init__
   self.LoginMFA(username, password, google_credential_file)
 File "/Users/username/Documents/Github/arlo-cloudflarefix/arlo.py", line 256, in LoginMFA
   ).execute()
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
   return wrapped(*args, **kwargs)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/http.py", line 923, in execute
   resp, content = _retry_request(
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/http.py", line 191, in _retry_request
   resp, content = http.request(uri, method, *args, **kwargs)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google_auth_httplib2.py", line 209, in request
   self.credentials.before_request(self._request, method, uri, request_headers)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/auth/credentials.py", line 134, in before_request
   self.refresh(request)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/oauth2/credentials.py", line 319, in refresh
   ) = reauth.refresh_grant(
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/oauth2/reauth.py", line 349, in refresh_grant
   _client._handle_error_response(response_data, retryable_error)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/oauth2/_client.py", line 69, in _handle_error_response
   raise exceptions.RefreshError(
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})
2023-10-13 13:21:09,783 | INFO | Function: main() | Line 89 | 

ok never mind, I just went through the google MFA steps again and re-created the credentials and now it works!!!! wow, your PR should def get merged into the master! thank you so much for the fix!

[cgmckeever]

Last I knew @jeffreydwalter was no longer using this library .. maybe time for a hand-off of fork?

[scuc]

Last I knew @jeffreydwalter was no longer using this library .. maybe time for a hand-off of fork?

yeah, I hope we can keep this library alive, I have a lot of money invested in Arlo cams. And all I really want to do is download my videos for a local archive, so in that regard, the arlo package has been been a huge help.