Confirm OAuth OOB and Implement Temporary Workaround

[atwater-tempus]

This morning (Sept 12) a PyDrive reliant script of ours failed. The full stacktrace is below. The script ran successfully last week, and there have been no changes to the configuration on our end. I think this is due to Google's impending removal of support for OAuth Out-of-Band flow.

Google's migration docs indicate that users can

Add an ack_oob_shutdown parameter with a value of the enforcement date: 2022-10-03 to your redirect flow request. Example: ack_oob_shutdown=2022-10-03

The Migrating to Google Auth Library PR is open. If it will not be merged in short order, I suggest implementing the temporary workaround.

Stacktrace

Traceback (most recent call last):
...
  File "/Users/.../lib/python3.8/site-packages/pydrive2/files.py", line 512, in Upload
    self._FilesInsert(param=param)
  File "/Users/.../lib/python3.8/site-packages/pydrive2/auth.py", line 84, in _decorated
    return decoratee(self, *args, **kwargs)
  File "/Users/.../lib/python3.8/site-packages/pydrive2/files.py", line 683, in _FilesInsert
    raise ApiRequestError(error)
pydrive2.files.ApiRequestError: <HttpError 403 when requesting https://www.googleapis.com/drive/v2/files?supportsAllDrives=true&alt=json returned "Access Not Configured. Drive API has not been used in project xxxxx before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/drive.googleapis.com/overview?project=xxxxx then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.". Details: "[{'domain': 'usageLimits', 'reason': 'accessNotConfigured', 'message': 'Access Not Configured. Drive API has not been used in project xxxxx before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/drive.googleapis.com/overview?project=xxxxx then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.', 'extendedHelp': 'https://console.developers.google.com/apis/api/drive.googleapis.com/overview?project=xxxxx'}]">

[shcheklein]

I suggest implementing the temporary workaround.

Sounds good. Would you be able to make a PR for this?

The https://github.com/iterative/PyDrive2/pull/221 PR is open. If it will not be merged in short order

This PR won't solve the problem though. The problem is that CLI auth method won't work anymore. After this PR is merged it'll be throwing an exception. You need to migrate either to the LocalWebserverAuth or service account.

[yeralexey]

Hi. Can it affect somehow on authentication with settings.yaml? I manage to get file info from google drive, but when I try to download the file with GetContentFile(), but receive "HttpError 403". My scopes seems to be fine, because when I download with manual authentication - I get what I need.

One the other hand, if my settings.yaml were incorrect, I would've been able to even get info about files with GetList() without manual, but I do...

[shcheklein]

@yeralexey settings.yaml is one of the possible ways to setup the PyDrive2 client, including the type of authentication to use. Could you give a bit more details / share your code? E.g. what do you mean by the manual authentication is not very clear.

HttpError 403

Could you publish the complete message you are getting?

[yeralexey]

@yeralexey settings.yaml is one of the possible ways to setup the PyDrive2 client, including the type of authentication to use. Could you give a bit more details / share your code? E.g. what do you mean by the manual authentication is not very clear.

HttpError 403

Could you publish the complete message you are getting?

@shcheklein

The code i am using is:

    fileid = "15********tX"
    filename = "IMG_20210303_165511.jpg"
    mimetype = "image/jpeg"

    file = go.CreateFile({'id': fileid})
    file.GetContentFile(f"{save_path}/{filename}", mimetype=mimetype, chunksize=26214400)
    pdb.set_trace()

By manual authentification i mean when i have no "settings.yaml" file in my project folder, only "client_secrets.json". So browser gets opened on page, where i need to confirm authentification by clicking. Download works fine, unless mimetype of the file is not "application/vnd.google-apps.document", in this case i get "The requested conversion is not supported. [400]" ))))

When i run it with this exaple, configured with my credentials data and with credentials.json in project folder appeared (so "save_credentials: True" in settings file) i get this:

  File "C:\PyProj\gotoya\venv\lib\site-packages\pydrive2\files.py", line 338, in GetContentFile
    download(
  File "C:\PyProj\gotoya\venv\lib\site-packages\pydrive2\files.py", line 328, in download
    status, done = downloader.next_chunk()
  File "C:\PyProj\gotoya\venv\lib\site-packages\googleapiclient\_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "C:\PyProj\gotoya\venv\lib\site-packages\googleapiclient\http.py", line 780, in next_chunk
    raise HttpError(resp, content, uri=self._uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/drive/v2/files/15Gzh****aUzJtX?acknowledgeAbuse=false&alt=media returned "The authenticated user has not granted the app 15****2 read access to the file 15G*******aUzJtX". Details: "[{'domain': 'global', 'reason': 'fileAccess', 'message': 'The authenticated user has not granted the app 15***282 read access to the file 15Gz****UzJtX', 'locationType': 'header', 'location': 'Authorization'}]">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\PyProj\gotoya\main.py", line 219, in <module>
    main()
  File "C:\PyProj\gotoya\main.py", line 214, in main
    gotoya(filedata)
  File "C:\PyProj\gotoya\main.py", line 190, in gotoya
    getfile(item, filedata['root']['files'][item][0], filedata['root']['files'][item][2])
  File "C:\PyProj\gotoya\main.py", line 54, in getfile
    file.GetContentFile(f"{save_path}/{filename}", mimetype=mimetype, chunksize=26214400)
  File "C:\PyProj\gotoya\venv\lib\site-packages\pydrive2\auth.py", line 84, in _decorated
    return decoratee(self, *args, **kwargs)
  File "C:\PyProj\gotoya\venv\lib\site-packages\pydrive2\files.py", line 350, in GetContentFile
    raise exc
pydrive2.files.ApiRequestError: <HttpError 403 when requesting https://www.googleapis.com/drive/v2/files/15Gzh***Di3aUzJtX?acknowledgeAbuse=false&alt=media returned "The authenticated user has not granted the app 150****282 read access to the file 15Gzhl***i3aUzJtX". Details: "[{'domain': 'global', 'reason': 'fileAccess', 'message': 'The authenticated user has not granted the app 150*82 read access to the file 15G***tX', 'locationType': 'header', 'location': 'Authorization'}]">

[andygikling]

Bump @shcheklein

First of all thank you so much for all our work on this library.

I have a few questions about this OOB issue I was hoping you could comment on.

I've been tasked with correcting this OOB situation at my company. We're using this library on a self hosted Ubuntu 22 Server build server that receives Github Actions Workflows jobs. Once our artifacts are compiled we LOVE using pydrive2 to upload them to GoogleDrive! Thanks again.

When I test the upload script manually on my desktop machine when using the gauth.CommandLineAuth() the browser window on my desktop pops up the OAuth window from Google and it's very clear that this method is no longer supported. I see this message in by browser before accepting the token:

"To help protect your account, Google will soon block apps that don’t comply with Google’s security policies. Make sure you are using the latest version of this app. You can let the app developer () know that this app should stop using unsupported OAuth flows."

When I change my upload script to use gauth.LocalWebserverAuth() as you suggested, I am not presented with the message about unsupported OAuth flows when presented with the acceptance page in my browser. Great.

My question then is how is gauth.LocalWebserverAuth() supposed to work on headless systems with no browser?

If I needed a new Actions self hosted runner which is running Ubuntu 22 Server - there is no desktop, there is no browser. This wasn't a problem with CommandlineAuth() because you can use a different browser to get the authorization number and paste that into the terminal on the headless Ubuntu Server.

Can you make a suggestion as to how one might get a credentials for pydrive2 on a headless server?

A note to people reading this. As long as your Client App's OAuth2 "type" is "Desktop" the OOB issue doesn't appear to have anything to do with needing an updated token of a different type. You just need to use gauth.LocalWebserverAuth() as @shcheklein pointed out at the beginning of this thread.

[shcheklein]

My question then is how is gauth.LocalWebserverAuth() supposed to work on headless systems with no browser?

How do you run the command line one btw? You have to SSH then from time to time, right? If you can do that then may be you can setup an SSH tunel from a local machine to remote and open that link locally?

Can you make a suggestion as to how one might get a credentials for pydrive2 on a headless server?

Alternative is the use of the service account flow + delegation to mitigate some limitations (e.g. service account by default creates files in its "domain" and has space constraints (1 2)

[andygikling]

Thanks for the rapid response!

Correct. I have access to the self-hosted runners via SSH all the time. An SSH port redirect tunnel is a very clever solution to my problem - I didn't even think of it! I'll try that.

Seems like the "right" solution for build automation servers is to move to a proper service account, thanks for clarifying. Any idea how one loads those credentials into pydrive2? I'll take a look.

[shcheklein]

Seems like the "right" solution for build automation servers is to move to a proper service account, thanks for clarifying. Any idea how one loads those credentials into pydrive2? I'll take a look.

I think if it's about automation the best way is to use an ENV variable, read it in the code and pass as json string. PyDrive2 init supports that.