OAuth 2.0 Credential Format for Delta Sharing Client

zhuansunxt commented 1 year ago

This is a proposal to add OAuth2.0 support in Delta Sharing.

2-legged OAuth for Delta Sharing

We will focus on supporting 2-legged OAuth flow for Delta Sharing which is suitable for programmatic applications like Apache Spark and Pandas. In a 2-legged OAuth flow, the client application will interact with an Authorization Server (AS) to obtain its OAuth credentials and use that credential to authenticate to the Resource Server (RS) to access data. In the Delta Sharing case, AS can be any 3rd party service hosted by identity provider and RS is the Delta Sharing Server hosting the data. Compared to 3-legged OAuth, no end user's intervention is needed and no end user credential is involved.

Token Endpoint on Authorization Server (AS)

RFC 8693 fully specifies the standard and best practice of an OAuth token exchange endpoint. When configured to use OAuth, A Delta Sharing connector as client application is going to assume an available token exchange endpoint that implements the Client Credentials Grant flow of the OAuth 2.0 specification. Specifically:

The server takes a HTTP POST request to exchange OAuth client ID and client secret for an access token
Authorization HTTP header is accepted to pass client application credentials.
- The client_id and client_secret are used in the ‘Basic’ HTTP Authentication scheme (RFC 7617) to compute the value of the Authorization HTTP header.
- Its value is composed of the keyword “Basic”, followed by a space and the base64 encoding of the concatenation of your client_id, “:” and your client_secret.
Accept HTTP header is accepted with applicaton/json value.
Content-Type HTTP header is accepted with application/x-www-form-urlencoded value.
grant_type in post body is accepted with client_credentials value.

The simplest way to request access token is by making below CURL request:

curl -X POST \
--
  | -H "Authorization: Basic $(echo -n $CLIENT_ID:$CLIENT_SECRET | base64)" \
  | -H "Content-Type: application/x-www-form-urlencoded" \
  | -H "Accept: application/json" \
  | -d "grant_type=client_credentials" \
  | https://example.api.com/oauth/v1/token

The format for Client Application Credentials File

We propose the below format of a credential file for Delta Sharing connector to accept for OAuth configuration.

File format: json

Field Name	Descrption
shareCredentialsVersion	The format version of the credential file. Use `2` for OAuth.
type	The type of the authentication. Use `oauth_client_credentials` (`persistent_oauth`deprecated) for 2-legged, programmatic OAuth.
endpoint	The url of the sharing server.
tokenEndpoint	The url of the OAuth2.0 token exchange endpoint
clientId	An OAuth client identifier.
clientSecret	An OAuth client secret.

Example:

{
  "shareCredentialsVersion": 2,
  "type": "oauth_client_credentials",
  "endpoint": "https://sharing.delta.io/delta-sharing",
  "tokenEndpoint": "https://example.api.com/oauth/v1/token",
  "clientId": "3896f8e6-cccc-4fc4-a342-59274b8a5e65",
  "clientSecret": "<redacted>"
}

The client that is configured with credential file should use the token endpoint to get access token and use it to authenticate to the sharing server.

zhuansunxt commented 1 year ago

This format is designed to be future-proof and extensible for more authentication schemes in the future. The type field will decide which authentication method to use and the connector should retrieve the corresponding fields from the json object. Below are two examples on how this format should work for the existing bearer token authentication and a to-be-added HTTP basic username:password auth.

Bearer token auth

{
  "shareCredentialsVersion": 2,
  "type": "bearer_token",
  "endpoint": "https://sharing.delta.io/delta-sharing",
  "bearerToken": "<token>",
  "expirationTime": "2021-11-12T00:12:29.0Z"
}

HTTP Basic Auth

{
  "shareCredentialsVersion": 2,
  "type": "basic",
  "endpoint": "https://sharing.delta.io/delta-sharing",
  "username": "alice",
  "password": "<redacted>"
}

zhuansunxt commented 1 year ago

This proposal is a result of discussion with @davidgreenfield , @chakankardb and @wchau.

dialberg commented 1 year ago

Hi,

Could you please review and approve the discussed issue code fix located at :

https://github.com/dialberg/delta-sharing

The current fix does not change the current system’s behavior and functionality and adds bearer token auth and basic authentication schemes.

All ../python/tests passed successfully.

We would be very grateful to get the approval to perform the pull request and see implemented changes in the client.

Thanks, Dima

zhuansunxt commented 1 year ago

Hi @dialberg , is there a single PR or commit that has all the changes you need a review for? I went through the commit history of yours and figured most changes should be in https://github.com/dialberg/delta-sharing/commit/e57cc89dc8912619d980b75de24c1605a664af2c. Can you confirm that?

Feel free to start a pull request against this repo. It'll make the review easier and merge process faster.

dialberg commented 1 year ago

Hi Xiaotong,

Yes, I confirm.

Thanks, Dima

From: Xiaotong Sun @.> Sent: יום ה 11 מאי 2023 21:24 To: delta-io/delta-sharing @.> Cc: Dima Alberg @.>; Mention @.> Subject: [External] : Re: [delta-io/delta-sharing] OAuth 2.0 Credential Format for Delta Sharing Client (Issue #269)

Hi @dialberghttps://urldefense.com/v3/__https:/github.com/dialberg__;!!ACWV5N9M2RV99hQ!MeKjV2Ooqy5PpxyUkdib8wcQuPOymXmxFbAtkbLlQgw1F0I9neO7Gfz2uIoFOKMdlgDDRTlPYCMNDK1rBkRfakYS-H-H$ , is there a single PR or commit that has all the changes you need a review for? I went through the commit history of yours and figured most changes should be in @.***https://urldefense.com/v3/__https:/github.com/dialberg/delta-sharing/commit/e57cc89dc8912619d980b75de24c1605a664af2c__;!!ACWV5N9M2RV99hQ!MeKjV2Ooqy5PpxyUkdib8wcQuPOymXmxFbAtkbLlQgw1F0I9neO7Gfz2uIoFOKMdlgDDRTlPYCMNDK1rBkRfam28JKy-$. Can you confirm that?

Feel free to start a pull request against this repo. It'll make the review easier and merge process faster.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/delta-io/delta-sharing/issues/269*issuecomment-1544477769__;Iw!!ACWV5N9M2RV99hQ!MeKjV2Ooqy5PpxyUkdib8wcQuPOymXmxFbAtkbLlQgw1F0I9neO7Gfz2uIoFOKMdlgDDRTlPYCMNDK1rBkRfav41rRzy$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A6ICHEPPE2P246KAL6BZC2TXFUVDRANCNFSM6AAAAAAU4P2OBM__;!!ACWV5N9M2RV99hQ!MeKjV2Ooqy5PpxyUkdib8wcQuPOymXmxFbAtkbLlQgw1F0I9neO7Gfz2uIoFOKMdlgDDRTlPYCMNDK1rBkRfagCuRMpu$. You are receiving this because you were mentioned.Message ID: @.**@.>>

dialberg commented 1 year ago

Hi @zhuansunxt,

Pull request branch with one commit was created. Branch : dialberg_delta_sharing_python_client_pull_request_12M23 Commit : dialberg@c0c5522

Thanks, Dima

zhuansunxt commented 1 year ago

Hi @dialberg,

That commit you linked does not look quite right. It is a change about pre-signed URL caching. Can you double check and link to the right commit that you are looking for a review?

dialberg commented 1 year ago

Hi @zhuansunxt,

Sorry. Updated. Branch : dialberg_delta_sharing_client Commit : dialberg/delta-sharing@33bd3ba Is it correct now?

Thanks, Dima

dialberg commented 1 year ago

Hi @zhuansunxt,

Could I open PR against 'main' base branch ?

Thanks, Dima

moderakh commented 1 month ago

we are changing persistent_oauth2.0 to oauth_client_credentials

delta-io / delta-sharing