[Discussion] Storage and locking

[theacodes]

(post-1.0.0 discussion of implementing credential storage and locking)

/cc @bshaffer

[Mackey22]

@jonparrott I need storage capability for a project I'm working on that needs to be code-complete by the end of next week (8/11/2017). https://github.com/GoogleCloudPlatform/google-auth-library-python/pull/165/files says that oauth2client will still be functional for the foreseeable future - I just want to make sure that's still the case before going ahead and using that. Thanks

[theacodes]

that is indeed the case. oauth2client is deprecated but we have no plans to remove it.

[amit-am]

Does this mean, "google-api-python-client" would be replaced by something else ? Is "google-api-python-client" library under development ? Will there be support as and when Google-Cloud has more features adding to its services & that incorporating into "google-api-python-client" library ?

[theacodes]

Does this mean, "google-api-python-client" would be replaced by something else ?

Eventually yes, but in the near-term no.

Is "google-api-python-client" library under development ?

It is in maintenance mode - we will fix bugs but we will not be adding any new features.

[amit-am]

Eventually yes, but in the near-term no.

My current code infra depends on this library. Should i consider migrating to another library which would be there long term ? Which is the other library that i should think of moving to ?

It is in maintenance mode - we will fix bugs but we will not be adding any new features.

I've been using google-api-python-client library & it's not thread-safe. So the library that you'll mention in answer to above question will be thread-safe ?

[theacodes]

My current code infra depends on this library. Should i consider migrating to another library which would be there long term ? Which is the other library that i should think of moving to ?

We will continue to address issues in google-api-python-client until we formally deprecate it and then a year after that. We do not have any plans to do right now or any timeline. For cloud APIs, many of the Cloud Client Libraries are now GA and we suggest using those for new projects.

I've been using google-api-python-client library & it's not thread-safe. So the library that you'll mention in answer to above question will be thread-safe ?

The Cloud Client Libraries are thread-safe.

While I appreciate these questions and I'm happy to answer, this is the wrong project and issue to ask. I'd suggest filing issues on https://github.com/google/google-api-python-client.

[Dave-Snigier]

I have begun using this library on a "serverless" project where I'm making use of Google services and APIs in programs that exist within ephemeral containers. Storage would be helpful as I'm concerned there may be thousands of access tokens for a refresh token at any given time, eventually hitting a limit.

I'm not super familiar with oauth flows and architecture, but here are a few things that come to my mind:

• If all clients try to refresh the token at the same time we run into a thundering herd situation
• Ideally would only work with the central store when the token is near expiration or invalid
• High-level storage approach should work with anything from a relational DB with row locking to an eventually-consistant key-value store

Perhaps something along these lines?

if expiry - random(5-10 minutes) < Now()
    db = get info from database

    if db.expiry < random expiry
        assume someone else got there first and update credentials from this database data
    else
        open transaction/aquire lock (if applicable)
        refresh token
        update database and release lock/commit

I think this is a happy compromise between collisions, load on the database, and load on the authentication server. The 5 to 10 minutes I pulled out of the air, with the lower bound being your choice of 5 minutes clock skew (PS I agree with that choice as it's what MIT kerberos and Microsoft AD Kerberos by default... a lot of people are used to that number). I'm sure we could put a bit more rigor into finding the upper bound.

I do like the idea of a more generic interface for storage, however, if this is going to be part of the first-run experience working with Google APIs on one of the most popular beginner languages, perhaps there should be some batteries included for redis and certainly for cloud memory store. Let people like me contribute the AWS plugins.

Thoughts?

[sduskis]

@theacodes, is this issue still relevant for the google-auth-library-python repo?

[theacodes]

It's a discussion and we don't have storage and locking support, so yes.

On Thu, Dec 6, 2018, 10:16 AM Solomon Duskis notifications@github.com wrote:

@theacodes https://github.com/theacodes, is this issue still relevant for the google-auth-library-python repo?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/googleapis/google-auth-library-python/issues/33#issuecomment-444974507, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPUc0C63m6tBfZTdaywlJEHJqEkh-ioks5u2V8RgaJpZM4KaWLJ .

[ei-grad]

It doesn't look like a big deal to implement the storage outside of google.auth, but it is a bit unclear how to store the credentials object (which fields to store? there is no universal interface to serialize/deserialaze it... one can use the pickle, or hardcode the token/expiry fields and use json, but...) and how to integrate the storage with their refresh cycle.

[theacodes]

Just all of the public properties:

https://google-auth.readthedocs.io/en/latest/reference/google.oauth2.credentials.html

See https://github.com/googleapis/google-auth-library-python-oauthlib/blob/master/google_auth_oauthlib/tool/__main__.py#L108

On Wed, May 22, 2019 at 4:24 AM Andrew Grigorev notifications@github.com wrote:

It doesn't look like a big deal to implement the storage outside of google.auth, but it is a bit unclear how to store the credentials object (which fields to store? there is no explicit way to serialize it, besides pickle of course, but ... err...) and how to integrate the storage with their refresh cycle.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/googleapis/google-auth-library-python/issues/33?email_source=notifications&email_token=AAB5I46TUDPGVB3NHWANGRTPWUUNHA5CNFSM4CTJMLE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV6XPRY#issuecomment-494761927, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB5I42IGBF32WYQAQ7AK6TPWUUNHANCNFSM4CTJMLEQ .

[ei-grad]

Thanks! The evil Flow.credentials property wrapper dissembled them from me in the IPython shell, ha-ha :-D.

I'll just put it here for clearness :-).

import datetime
import google.auth
import google.oauth2
import google_auth_oauthlib.flow

flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file( 
    'client_secret.json',
    scopes=["https://www.googleapis.com/auth/userinfo.email",
            "https://www.googleapis.com/auth/userinfo.profile"],
) 
flow.run_local_server()  # or flow.run_console()
session = flow.authorized_session()
print(session.get('https://www.googleapis.com/userinfo/v2/me').json())

creds = flow.credentials

info = { 
    'token': creds.token,
    'refresh_token': creds.refresh_token,
    'token_uri': creds.token_uri,
    'client_id': creds.client_id,
    'client_secret': creds.client_secret,
    'scopes': creds.scopes,
    'expiry': creds.expiry.isoformat(),
}

# implement your storage logic here, e.g. just good old json.dump() / json.load()

creds = google.oauth2.credentials.Credentials( 
    token=info['token'],
    refresh_token=info['refresh_token'],
    token_uri=info['token_uri'],
    client_id=info['client_id'],
    client_secret=info['client_secret'],
    scopes=info['scopes'],
)
creds.expiry = datetime.datetime.fromisoformat(info['expiry'])

if creds.expired:
    # don't forget to dump one more time after the refresh
    # also, some file-locking routines wouldn't be needless
    creds.refresh(google.auth.transport.requests.Request())

Any remarks?

[jmdobry]

Test comment (plz ignore, will delete in a bit)

[westarle]

Closing this very old discussion.