Open liufuyang opened 1 year ago
Hi @liufuyang did you manage to solve this problem? I'm thinking it could have something to do with newer version of gcloud? I'll investigate
I am not quite sure. there is no field of private_key
in the json file.
Hi, @liufuyang did you fix the issue?
I successfully use gcp_auth
to fetch the token based on the following env
Based on my testing, I am able to decrypt based on the token from gcp_auth
in the following environments.
To see if there is any difference from yours, I'd love to check this up. Or, you could provide a repository that reproduces the issue for me. Thanks 🙏🏻
Thanks a lot. Very sorry I lost my environment when I tested this and tried again with the setup and it works fine.
But I remember previously I was playing with gcloud auth application-default loginlogin --impersonate-service-account=<Service Account>
, perhaps I had a GOOGLE_APPLICATION_CREDENTIALS
pointing to another json file when I did the test.
So I tried again to generate a json key with the command above, using --impersonate-service-account
to bind an SA.
Then the generated application_default_credentials.json
looks like
{
"delegates": [],
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/xxx@xxx.iam.gserviceaccount.com:generateAccessToken",
"source_credentials": {
"client_id": "xxxxx.apps.googleusercontent.com",
"client_secret": "xxxxx",
"refresh_token": "xxxxx",
"type": "authorized_user"
},
"type": "impersonated_service_account"
}
And when using this impersonated_service_account
type key it gives this
Error: CustomServiceAccountCredentials(Error("missing field
private_key", line: 11, column: 1))
Perhaps you can take a look in this direction? Maybe it is nice to be able to support impersonated_service_account? Otherwise, it is not anything urgent as it might be a future not many people would use in practice.
I'm sorry for not getting back to you sooner. I was too busy to check this up for you lately. After checking, I find this crate should be able to handle your case.
If not, I think it's better to update the crate to the latest version
.
Below are the steps on how I tested it.
gcloud auth application-default login --impersonate-service-account=<service-account-email>
referencecat $HOME/.config/gcloud/application_default_credentials.json
which is the same as what you provided withcargo test
gcloud SDK INFO -- Google Cloud SDK 424.0.0
ALL TESTs passed.
Apart from that, I have built a tiny script to confirm the functionality.
Please don't hesitate to check this by yourself.
cargo add reqwest
touch src/main.rs
use reqwest::header::CONTENT_TYPE;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let authentication_manager = gcp_auth::AuthenticationManager::new().await?;
let _token = authentication_manager
.get_token(&["https://www.googleapis.com/auth/cloud-platform"])
.await?;
// Ref: https://cloud.google.com/storage/docs/listing-objects#permissions-rest
let bucket_name = "REPLACE_THIS_BY_YOURS_BUCKET_NAME".to_string();
let client = reqwest::Client::new();
let uri = format!(
"https://storage.googleapis.com/storage/v1/b/{}/o",
bucket_name
);
let response = client.get(uri)
.header(CONTENT_TYPE, "application/x-www-form-urlencoded")
.bearer_auth(&_token.as_str().to_string()).send().await?.text().await?;
println!("Response: {}", response);
Ok(())
}
thanks @dacozai for a suggestion, however I also see value in having direct support for impersonation so I'll leave this open so far
@liufuyang May I ask which version you use? gcp_auth: ? gcloud SDK: ?
thanks 🙏🏻
Google Cloud SDK 406.0.0 gcp_auth: 0.8.0
@dacozai Okay, it seems a bug (or just missing a feature) in the code when the GOOGLE_APPLICATION_CREDENTIALS
is used for an authorised_user
or impersonated_service_account
type of JSON key. I did it like this:
gcloud auth application-default
(doesn't matter if impersonation is used or not)cat ~/.config/gcloud/application_default_credentials.json
, and it shows "type": "authorized_user"
cargo run
, and it works.GOOGLE_APPLICATION_CREDENTIALS=/Users/fuyangl/.config/gcloud/application_default_credentials.json cargo run
, basically manually specify the key to use, then it fails.
Error: CustomServiceAccountCredentials(Error("missing field 'private_key'", line: 6, column: 1))
If an impersonation account JSON is used, then the error shows as a different line 11: Error: CustomServiceAccountCredentials(Error("missing field 'private_key'", line: 11, column: 1))
My testing code is:
use gcp_auth::AuthenticationManager;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let authentication_manager = gcp_auth::AuthenticationManager::new().await?;
let token = authentication_manager
.get_token(&["https://www.googleapis.com/auth/cloud-platform"])
.await?;
println!("{}", token.as_str());
Ok(())
}
[dependencies]
gcp_auth = "0.8.0"
tokio = {version = "1.26.0", features = ["macros", "parking_lot", "rt-multi-thread"]}
So I guess if you just run with cargo run
or perhaps cargo test
, it doesn't use the /Users/fuyangl/.config/gcloud/application_default_credentials.json
file and then uses the gcloud
command, as mentioned in the main Readme?
This isn't restricted to impersonated service accounts. I am also seeing this with just using the standard application_default_credentials
brew install --cask google-cloud-sdk
gcloud -V
"Google Cloud SDK 441.0.0"gcloud auth application-default login
GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/application_default_credentials.json cargo run
0: authorizer error: Application profile provided in `GOOGLE_APPLICATION_CREDENTIALS` was not parsable
1: Application profile provided in `GOOGLE_APPLICATION_CREDENTIALS` was not parsable
2: missing field `private_key` at line 7 column 1', src/main.rs:18:34
~/.config/gcloud/application_default_credentials.json
{
"client_id": "*******.apps.googleusercontent.com",
"client_secret": "*******",
"quota_project_id": "*******",
"refresh_token": "*******",
"type": "authorized_user"
}
The reason I need to use GOOGLE_APPLICATION_CREDENTIALS instead of other methods is because I am trying to run my program locally under docker-compose where I won't have gcloud sdk installed.
Looking into this further, I found a stack overflow posts from back in 2019 that reference the application_default_credentials matching the format I am seeing, so this isn't an issue with a gcloud sdk update.
The problem is that gcp_auth doesn't support using GOOGLE_APPLICATION_CREDENTIALS to reference application_default_credentials.json
. Instead it requires that application_default_credentials.json
is found at $HOME/.config/gcloud/application_default_credentials.json
.
This behavior is contrary to other google tools. For example the cloud_sql_proxy handles GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/application_default_credentials.json
just fine.
I believe this issue should be split into 2 issues
GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/application_default_credentials.json
For others seeing this error due to docker, I can work around this by creating a non-root user with a home dir at /app
and then mounting the host's ~/.config/gcloud
to the guests /app/.config/gcloud
Okay, so it appears there are (at least) two different kinds of JSON being used, one perhaps for users and one for service accounts? The one for service accounts includes a private_key
whereas the other one does not.
In order to get this right, it would be good if someone could dig out references to Google documentation (or, to Google source code, for example in their SDKs for other languages) that more clearly shows how these differ and when we're supposed to use one vs the other.
@djc Thanks. Perhaps following some idea illustrated here? https://github.com/golang/oauth2/blob/ac6658e9cb5802cebf9b8fd5f5d58f22bedb527f/google/google.go#L160-L163
Here you also see the private_key
is used for Service Account
but some clientSecret
is used instead of that for User Credential (which typically comes from gcloud auth)
https://github.com/golang/oauth2/blob/ac6658e9cb5802cebf9b8fd5f5d58f22bedb527f/google/google.go#L108-L120
Maybe the right fix here is to make the AuthenticationManager
check for GOOGLE_APPLICATION_CREDENTIALS
first (before trying all the different implementations) and, if it exists and contains a valid path, try to deserialize the contents as either the current ApplicationCredentials
or UserCredentials
(using a serde
untagged
annotation) and then proactively try to work with that?
I'm not great at reading Go code but it seems like that might be more in line with what they do there.
@valkum this might be of interest to you.
Using GOOGLE_APPLICATION_CREDENTIALS
should already be handled. If there are some issues there seems to be a bug with the implementation of CustomServiceAccount
.
I see some fields in the go implementation that are missing in ApplicationCredentials
Looking at the go implementation the file can have different formats: It tries the following format first
type cred struct {
ClientID string `json:"client_id"`
ClientSecret string `json:"client_secret"`
RedirectURIs []string `json:"redirect_uris"`
AuthURI string `json:"auth_uri"`
TokenURI string `json:"token_uri"`
}
var j struct {
Web *cred `json:"web"`
Installed *cred `json:"installed"`
}
And if that fails, it tries this one:
type credentialsFile struct {
Type string `json:"type"`
// Service Account fields
ClientEmail string `json:"client_email"`
PrivateKeyID string `json:"private_key_id"`
PrivateKey string `json:"private_key"`
AuthURL string `json:"auth_uri"`
TokenURL string `json:"token_uri"`
ProjectID string `json:"project_id"`
// User Credential fields
// (These typically come from gcloud auth.)
ClientSecret string `json:"client_secret"`
ClientID string `json:"client_id"`
RefreshToken string `json:"refresh_token"`
// External Account fields
Audience string `json:"audience"`
SubjectTokenType string `json:"subject_token_type"`
TokenURLExternal string `json:"token_url"`
TokenInfoURL string `json:"token_info_url"`
ServiceAccountImpersonationURL string `json:"service_account_impersonation_url"`
ServiceAccountImpersonation serviceAccountImpersonationInfo `json:"service_account_impersonation"`
Delegates []string `json:"delegates"`
CredentialSource externalaccount.CredentialSource `json:"credential_source"`
QuotaProjectID string `json:"quota_project_id"`
WorkforcePoolUserProject string `json:"workforce_pool_user_project"`
// Service account impersonation
SourceCredentials *credentialsFile `json:"source_credentials"`
}
type serviceAccountImpersonationInfo struct {
TokenLifetimeSeconds int `json:"token_lifetime_seconds"`
}
We should update our implementation here to match the structs used in the official sdk. The go code and the python code also distinguish the content of the file loaded via the env var. See here for go.. This logic might be missing in CustomServiceAccount
.
Using the go version might need some more reverse engineering to know which JSON fields are optional.
I don't think we should list all the fields we don't need here -- serde will ignore fields we don't need by default, which seems right for this. I think the way we should structure the data types here is to have an untagged enum to separate out what the Go code calls "Service Account fields" from the "User Credential fields". However, to the extent the token updating routines would be different for service account vs user credentials, it probably still makes sense to leave those in the two different trait implementations while extracting the code that inspects the GOOGLE_APPLICATION_CREDENTIALS
environment variable value.
I think the enum approach from @djc sounds the most straightforward and idiomatic.
But I think we could do it with a tagged enum instead of an untagged enum. Looking at the google code, they discriminate on the type
field, which we could get serde to do with #[serde(tag = "type")]
. Then we would need to alias or rename the variant names to match the ones google users with #[serde(alias = "<tag_name>")]
.
Looking over the go reference code, the tag names used are these:
service_account
for the current service account private_key approachauthorized_user
for application_default_credentials.json
And with this approach, it would be very straightforward to extend the enum to support the other credential methods listed in the go reference
impersonated_service_account
for an impersonated service account. Which is what this issue was originally asking aboutexternal_account
for an external account (not sure what this means)The struct is internally tagged by the type
field which should allow us to get a clean representation of the different types during parsing.
A quick pass over tokenSource
yields the following needed representation:
enum Credentials {
ServiceAccount(ServiceAccountCredentials),
UserCredentials(UserCredentials),
ExternalAccount(ExternalAccountCredentials),
ImpersonateServiceAccount(ImpersonateServiceAccountCredentials)
}
struct ServiceAccountCredentials {
client_email: String,
private_key: String,
private_key_id: String,
token_uri: Option<String>,
audience: String
}
// implementation to turn ServiceAccountCredentials into some kind of common config form.
// Needs optional `scopes` and optional `subject` user to impersonate.
// Replaces `token_url` with fallback.
// Refresh logic: https://github.com/golang/oauth2/blob/2e4a4e2bfb69ca7609cb423438c55caa131431c1/jwt/jwt.go#L101
struct UserCredentials {
client_id: String,
client_secret: String,
auth_uri: Option<String>,
token_uri: Option<String>,
refresh_token: String
}
// implementation to turn UserCredentials into some kind of common config form
// Needs optional `scopes`
// Replaces `auth_url` and `token_url` with fallback
// Refresh logic: https://github.com/golang/oauth2/blob/master/oauth2.go#L269
struct ExternalAccountCredentials {
audience: String,
subject_token_type: String,
token_url: String, // Note: this is not token_uri from the other types.
token_info_url: String,
service_account_impersonation_url: Option<String>,
service_account_impersonation: ServiceTokenImpersonationInfo
client_secret: String,
client_id: String,
credential_source: CredentialSource,
quota_project_id: String,
workforce_pool_user_project: Option<String>
}
struct ServiceTokenImpersonationInfo {
token_lifetime_seconds: i64
}
// Omitting CredentialSource for now.
struct ImpersonateServiceAccountCredentials {
// Either an untagged enum or this
service_account_impersonation_url: Option<String>,
credential_source: Option<CredentialSource>,
delegates: Vec<String>
}
The Options are based on == ""
and != ""
checks. The used golang JSON deserializer falls back to a default of "" if a key is not present in the parsed JSON. I am not sure if they are omitted server side. I guess it would be reasonable to rely on Strings and also check for empty strings instead. There might be more fields for which this is true.
All of the fields from above are used here.
Edit: Sorry for pointing out similar things @msdrigg. Had this in my editor while looking through the go implementation.
Fair that we can use a tagged enum here. @msdrigg would you be able to take a shot at a PR?
I certainly would. I will take a crack at it in the next couple days or this weekend at the latest.
Awesome, thanks!
I have a PR that should fit all use cases except ExternalAccountCredentials. I need to test it but I would love anyone interested here to take a look and offer any feedback they can.
Hi there, thanks for creating this nice package.
Does it support all kinds of different key jsons.
I tried to use the the json generated via
gcloud auth application-default login
and the key is located at/Users/fuyangl/.config/gcloud/application_default_credentials.json
The key as "client_id", "client_secret", "refresh_token" and "type" as "authorized_user", that is all the fields there.