search

aws / aws-sdk-java

The official AWS SDK for Java 1.x. The AWS SDK for Java 2.x is available here: https://github.com/aws/aws-sdk-java-v2/
https://aws.amazon.com/sdkforjava
Apache License 2.0
4.12k stars 2.83k forks source link

credentials leak risk in AWSSecretsManagerClient #2702

Closed jkndrkn closed 1 year ago

jkndrkn commented 2 years ago

Describe the bug

When malformed credentials are passed to com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.getSecretValue(), a

com.amazonaws.services.secretsmanager.model.AWSSecretsManagerException is raised that contains the malformed credential. If the malformed credential contains sensitive data, these are included in the Exception message and will be persisted to logs which presents a security risk. For example, the below log message was generated by our application when a JSON object containing an IAM profile, an access key ID, and a secret access key was passed-in by accident. The "****" characters were added by me after the fact to protect the integrity of this profile.

com.amazonaws.services.secretsmanager.model.AWSSecretsManagerException: '"AWS_ACCESS_KEY_ID":"AKIA****"' not a valid key=value pair (missing equal-sign) in Authorization header: 'AWS4-HMAC-SHA256 Credential={"username":"****","AWS_ACCESS_KEY_ID":"AKIA****","AWS_SECRET_ACCESS_KEY":"EjcX****"}/20220119/us-east-2/secretsmanager/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-date;x-amz-target, Signature=10f301176f075853adc9eeae0231c9855e724a37757553f86f314e12a8aa3bed'. (Service: AWSSecretsManager; Status Code: 400; Error Code: IncompleteSignatureException; Request ID: 15db2412-ade7-48d6-b5bf-7047ce3fdd0e; Proxy: null)

Expected behavior

If malformed credentials are submitted, these should not be included in exception message.

Current behavior

Malformed credentials are emitted in the exception message.

com.amazonaws.services.secretsmanager.model.AWSSecretsManagerException: '"AWS_ACCESS_KEY_ID":"AKIA****"' not a valid key=value pair (missing equal-sign) in Authorization header: 'AWS4-HMAC-SHA256 Credential={"username":"****","AWS_ACCESS_KEY_ID":"AKIA****","AWS_SECRET_ACCESS_KEY":"EjcX****"}/20220119/us-east-2/secretsmanager/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-date;x-amz-target, Signature=10f301176f075853adc9eeae0231c9855e724a37757553f86f314e12a8aa3bed'. (Service: AWSSecretsManager; Status Code: 400; Error Code: IncompleteSignatureException; Request ID: 15db2412-ade7-48d6-b5bf-7047ce3fdd0e; Proxy: null)

Steps to Reproduce

Construct a AWSSecretsManagerClient.getSecretValue() request using the below string value for either the access key ID or the secret access key:

{"AWS_ACCESS_KEY_ID":"AKIA****","AWS_SECRET_ACCESS_KEY":"EjcX****"}

Possible Solution

Instead of printing the malformed credentials, simple state that the credentials are malformed. The user can then debug their application to see what values they are accidentally passing-in.

Context

Leaked credentials caused us to have to engage our IT staff to regenerate credentials and then notify our developers to update their credentials.

AWS Java SDK version used

1.11.968

JDK version used

openjdk version "11.0.10" 2021-01-19 OpenJDK Runtime Environment (build 11.0.10+9) OpenJDK 64-Bit Server VM (build 11.0.10+9, mixed mode)

Operating System and version

macOS 11.6.1

debora-ito commented 2 years ago

@jkndrkn thank you for reaching out.

I'm having trouble to reproduce. How exactly was the secret created in the first place?

The credentials that were logged were not the credentials returned by the getSecretValue operation but rather the credentials used to make that API call, is that correct?

Can you share a code sample showing how you are creating the SecretsManager client?

Can you provide the full stacktrace of the error? Please continue to redact the sensitive info as you've been doing in the description.

jkndrkn commented 2 years ago

Hi @debora-ito, thank you so much for your reply. We are using aws-sdk-java via the https://github.com/mcohen01/amazonica project. This is a Clojure project.

Here is the most relevant file in that project:

https://github.com/mcohen01/amazonica/blob/master/src/amazonica/aws/secretsmanager.clj

If you take a look at amz/set-client, you can see that the above file basically exposes all of the public methods of com.amazonaws.services.secretsmanager.AWSSecretsManagerClient as kebab-case Clojure functions.

The AWS Secrets Manager secret was created manually using the AWS Console. The contents is a JSON object.

The credentials that were logged were not the credentials returned by the getSecretValue operation but rather the credentials used to make that API call, is that correct?

Yes, that is correct.

Can you share a code sample showing how you are creating the SecretsManager client?

Here is a Clojure snippet that runs with no error. An important detail is that getSecretValue is passed a map of credentials and then a map containing the AWS Secrets Manager secret ID.

(let [secret-id "arn:aws:secretsmanager:us-east-2:****"
        credentials {:endpoint "us-east-2"
                     :profile "****"
                     :access-key "AKIA****"
                     :secret-key "****"}
        secrets (amazonica.aws.secretsmanager/get-secret-value credentials
                                                               {:secret-id secret-id})]
    (prn secrets))

Here is a snippet that does trigger the error. Note that a JSON string is passed to the :access-key parameter instead of the access key itself.

(let [secret-id "arn:aws:secretsmanager:us-east-2:****"
        credentials {:endpoint "us-east-2"
                     :profile "ai-labs-cxray-user"
                     :access-key "{\"AWS_ACCESS_KEY_ID\":\"AKIA****\",\"AWS_SECRET_ACCESS_KEY\":\"****\"}"
                     :secret-key "****"}
        secrets (amazonica.aws.secretsmanager/get-secret-value credentials
                                                               {:secret-id secret-id})]
    (prn secrets))

Can you provide the full stacktrace of the error? Please continue to redact the sensitive info as you've been doing in the description.

Happy to provide that. Keep in mind that this is a Clojure application so it can make understanding what is going on a bit tricky. However, you should be able to see the execution path through several com.amazonaws calls. Here is the stack trace produced by the above snippet of code that triggers the error:

; Execution error (AWSSecretsManagerException) at com.amazonaws.http.AmazonHttpClient$RequestExecutor/handleErrorResponse (AmazonHttpClient.java:1819).
; '"AWS_SECRET_ACCESS_KEY":"****"}/20220121/us-east-2/secretsmanager/aws4_request' not a valid key=value pair (missing equal-sign) in Authorization header: 'AWS4-HMAC-SHA256 Credential={"AWS_ACCESS_KEY_ID":"AKIA****","AWS_SECRET_ACCESS_KEY":"****"}/20220121/us-east-2/secretsmanager/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-date;x-amz-target, Signature=7be464fadd93840680201eb14d43536157a189064d6815bd769c67dbbfd1b548'. (Service: AWSSecretsManager; Status Code: 400; Error Code: IncompleteSignatureException; Request ID: efad3f15-aeea-46a6-b5f9-726e931c3c2c; Proxy: null)
com.amazonaws.http.AmazonHttpClient$RequestExecutor/handleErrorResponse (AmazonHttpClient.java:1819)
com.amazonaws.http.AmazonHttpClient$RequestExecutor/handleServiceErrorResponse (AmazonHttpClient.java:1403)
com.amazonaws.http.AmazonHttpClient$RequestExecutor/executeOneRequest (AmazonHttpClient.java:1372)
com.amazonaws.http.AmazonHttpClient$RequestExecutor/executeHelper (AmazonHttpClient.java:1145)
com.amazonaws.http.AmazonHttpClient$RequestExecutor/doExecute (AmazonHttpClient.java:802)
com.amazonaws.http.AmazonHttpClient$RequestExecutor/executeWithTimer (AmazonHttpClient.java:770)
com.amazonaws.http.AmazonHttpClient$RequestExecutor/execute (AmazonHttpClient.java:744)
com.amazonaws.http.AmazonHttpClient$RequestExecutor/access$500 (AmazonHttpClient.java:704)
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl/execute (AmazonHttpClient.java:686)
com.amazonaws.http.AmazonHttpClient/execute (AmazonHttpClient.java:550)
com.amazonaws.services.secretsmanager.AWSSecretsManagerClient/doInvoke (AWSSecretsManagerClient.java:3039)
com.amazonaws.services.secretsmanager.AWSSecretsManagerClient/invoke (AWSSecretsManagerClient.java:3006)
com.amazonaws.services.secretsmanager.AWSSecretsManagerClient/executeGetSecretValue (AWSSecretsManagerClient.java:1231)
com.amazonaws.services.secretsmanager.AWSSecretsManagerClient/getSecretValue (AWSSecretsManagerClient.java:1200)
jdk.internal.reflect.NativeMethodAccessorImpl/invoke0 (NativeMethodAccessorImpl.java:-2)
jdk.internal.reflect.NativeMethodAccessorImpl/invoke (NativeMethodAccessorImpl.java:62)
jdk.internal.reflect.DelegatingMethodAccessorImpl/invoke (DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method/invoke (Method.java:566)
amazonica.core/fn-call (core.clj:878)
amazonica.core/intern-function (core.clj:1065)
com.bamboohr.clj-utils.credentials/eval12502 (NO_SOURCE_FILE:127)
com.bamboohr.clj-utils.credentials/eval12502 (NO_SOURCE_FILE:121)
clojure.lang.Compiler/eval (Compiler.java:7181)
clojure.core/eval (core.clj:3202)
clojure.core/eval (core.clj:3198)
nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:87)
clojure.core/apply (core.clj:667)
clojure.core/with-bindings* (core.clj:1977)
nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:87)
clojure.main/repl (main.clj:437)
clojure.main/repl (main.clj:458)
clojure.main/repl (main.clj:368)
nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:84)
nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:56)
nrepl.middleware.interruptible-eval/interruptible-eval (interruptible_eval.clj:152)
nrepl.middleware.session/session-exec (session.clj:218)
nrepl.middleware.session/session-exec (session.clj:217)
java.lang.Thread/run (Thread.java:834)
debora-ito commented 1 year ago

@jkndrkn apologies for losing track of this issue. Thank you for the additional details.

When the service received the request, it expected to see a key=value pair for the access key, and printed the problematic values when it got something weird instead, to make it easier to identify the issue. In this case, it's unfortunate that the weird thing included the secret key.

What was printed was the request authorization header, in which normally would include the access key only, like:

"Authorization: AWS4-HMAC-SHA256 Credential=AKIA****/20230331/us-west-2/dynamodb/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-date;x-amz-security-token;x-amz-target, Signature=xxx[\r][\n]"

I don't think there's anything to fix in this case. Take a look at some of the best practices on how to properly manage aws access keys here: https://docs.aws.amazon.com/accounts/latest/reference/credentials-access-keys-best-practices.html#iam-user-access-keys