Support passing credential blob inline in GCS

pingcap / br

A command-line tool for distributed backup and restoration of the TiDB cluster data

https://pingcap.com/docs/dev/how-to/maintain/backup-and-restore/br/

Apache License 2.0

123 stars 102 forks source link

Support passing credential blob inline in GCS #635

Open kennytm opened 3 years ago

kennytm commented 3 years ago

Feature Request

Describe your feature request related problem:

Currently GCS requires a file to pass the credentials. Creating a file is not always possible, esp. in BR-in-SQL scenario.

Describe the feature you'd like:

Support passing the credentials directly as a URL parameter.

Option A: percent-escaping — gcs://bucket/path?credentials-json=%7B%22access_id%22%3A%22your_access_key_id%22%2C%22secret_key%22%3A%22your_secret_access_key%22%7D
Option B: base64-encoding — gcs://bucket/path?credentials-json=eyJhY2Nlc3NfaWQiOiJ5b3VyX2FjY2Vzc19rZXlfaWQiLCJzZWNyZXRfa2V5IjoieW91cl9zZWNyZXRfYWNjZXNzX2tleSJ9

Describe alternatives you've considered:

Rely entirely on #633.

Teachability, Documentation, Adoption, Migration Strategy:

IANTHEREAL commented 3 years ago

I prefer to add another argument in br and option in br via sql

iamxy commented 3 years ago

I prefer to provide a convenient solution for online encode, such as base64encode. So, Option B is good for me.

kennytm commented 3 years ago

If the user can create an HMAC key and set a default project, they can reuse the s3:// protocol as illustrated in https://cloud.google.com/storage/docs/migrating#migration-simple.

kennytm commented 3 years ago

The credentials blob is extremely long, see https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys for a template. So option B looks like the only viable way.

tennix commented 3 years ago

I think we can just postpone this feature, passing credentials around is not secure. The secure way is to grant permission directly to the ec2 node (iam role) or gce node (service account).

For TiDBCloud product, we can expose the underlying node's iam role or service account, then users just grant permissions to the iam role or service account.