When I use ProfileCredentialsProvider in the AWS Java SDK with credential_process, get issue "tls: failed to verify certificate: x509: OSStatus -26276"

[sxhmilyoyo]

Summary

I followed this awesome reference to set up the rolesanywhere on my MacBook: https://aws.amazon.com/blogs/security/extend-aws-iam-roles-to-workloads-outside-of-aws-with-iam-roles-anywhere/, and it works when I run AWS CLI on my MacBook. However, when I tested it in my code with AWS Java SDK in Spring Boot Application, it does not work.

~/.aws/config

[default]
region=us-west-2
credential_process = </path/to/aws_signing_helper> credential-process --certificate /path/to/cert.pem --private-key /path/to/private-key.pem --trust-anchor-arn <trust-anchor-arn> --profile-arn <profile-arn> --role-arn <role-arn>

Test with AWS CLI

Command

aws sts get-caller-identity

Response

{
    "UserId": "<userid-part1:userid-part2>",
    "Account": "<correct account id>",
    "Arn": "<correct assumed role arn>"
}

Test with AWS Java SDK in Spring Boot Application

Java Code

    public S3Client provideS3Client() throws InterruptedException {
        S3Client s3Client = S3Client.builder()
                .region(Region.US_WEST_2)
                .credentialsProvider(ProfileCredentialsProvider.create("default"))
                .crossRegionAccessEnabled(true)
                .build();
        return s3Client;

Error

Caused by: java.lang.IllegalStateException: Command returned non-zero exit value (1) with error message: 2024/05/02 22:24:13 RequestError: send request failed
caused by: Post "https://rolesanywhere.us-west-2.amazonaws.com/sessions?profileArn=arn%3Aaws%3Arolesanywhere%3Aus-west-2%3A<account-id>%3Aprofile%2F04da14fe-425d-45ea-babf-b508feebe73f&roleArn=arn%3Aaws%3Aiam%3A%3A<account-id>%3Arole%2Ftest-ats-outside-amazon&trustAnchorArn=arn%3Aaws%3Arolesanywhere%3Aus-west-2%3A<account-id>%3Atrust-anchor%2F2fa10e15-c6f6-4343-83a4-7588706335cc": tls: failed to verify certificate: x509: OSStatus -26276

Potential Fix

I noticed that the issue was from tls and there is one flag for aws_signing_helper: --no-verify-ssl. I tried to add --no-verify-ssl in the ~/.aws/config, like this

[default]
region=us-west-2
credential_process = </path/to/aws_signing_helper> credential-process --certificate /path/to/cert.pem --private-key /path/to/private-key.pem --trust-anchor-arn <trust-anchor-arn> --profile-arn <profile-arn> --role-arn <role-arn> --no-verify-ssl

Then no more issue, the S3Client started working with correct credentials.

Question

Could you help me understand the root cause of this issue? Besides, according to the official reference: https://docs.aws.amazon.com/rolesanywhere/latest/userguide/credential-helper.html#credential-helper-options

Important
Note that this disables TLS host authentication, and can open the connection to man-in-the-middle attacks. This option should only be used under specific, tightly controlled scenarios, such as debugging proxy connections.

It seems like it is better to not use --no-verify-ssl.

Thanks.

[13ajay]

I suspect that the trust store from which the Java SDK client is pulling CA certificates from is different from the one that the AWS CLI uses. I believe the Java SDK uses the default Java cacerts trust store, so you may be able to fix this by adding the Amazon Root CA certificate to your trust store.

[rlalcorn]

Two other avenues to investigate -

1) Are the two commands (CLI vs. Spring Boot) running on the same instance? The Java cacerts shouldn't affect the trust store the credential helper uses, but a different instance might.

2) Is the Spring Boot app launched with the environment variables HTTP_PROXY or HTTPS_PROXY set? The AWS Java SDK 2 ProcessCredentialsProvider (implicitly called by the ProfileCredentialsProvider when handling a profile with credential_process defined) uses the Java ProcessBuilder. According to the Javadoc, the child process is run with "copy of the environment of the current process" - so if the app is launched with those variables, I expect them to be in the credential helpers environment.

The Go standard library, used by the credential helper, will honor those variables. If set, it will connect to the proxy first, which presents a certificate - usually a private certificate that's not in the trust store. If you find this is the case, you can put the certificate into the trust store, or try launching the Boot app without the variables.

Independently, you can cerify the validity of the Roles Anywhere endpoint TLS certificate -

openssl s_client -connect rolesanywhere.us-west-2.amazonaws.com:443 -showcerts < /dev/null

You can also verify that the credential helper can access the endpoint with no TLS errors, when called directly -

aws_signing_helper credential-process --certificate /path/to/cert.pem --private-key /path/to/private-key.pem --trust-anchor-arn <trust-anchor-arn> --profile-arn <profile-arn> --role-arn <role-arn>

The key is figuring out what is different when the JVM forks out to the process.

[sxhmilyoyo]

@rlalcorn Thanks for your quick reply.

Are the two commands (CLI vs. Spring Boot) running on the same instance? The Java cacerts shouldn't affect the trust store the credential helper uses, but a different instance might.

Yes, they were both ran on my MacBook.

Is the Spring Boot app launched with the environment variables HTTP_PROXY or HTTPS_PROXY set?

I tried echo $HTTP_PROXY and echo $HTTPS_PROXY, nothing is returned. Besides, I also checked the application.properties file, nothing related to them either.

I tried to run aws_signing_helper credential-process --certificate /path/to/cert.pem --private-key /path/to/private-key.pem --trust-anchor-arn <trust-anchor-arn> --profile-arn <profile-arn> --role-arn <role-arn>, and it works for me.

I tried to run openssl s_client -connect rolesanywhere.us-west-2.amazonaws.com:443 -showcerts < /dev/null and it works as well for me to return some certificates and something like this in the end

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5574 bytes and written 403 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE

I am a newbie on the domain of the trust store and certificate. It would be appreciated if you could share more debug process with more details. Thanks.

[sxhmilyoyo]

@13ajay Thanks for your quick reply.

Let me explore on this.

I followed on this https://docs.aws.amazon.com/privateca/latest/userguide/PcaIssueCert.html to set up the end-entity certificates as required as the 2nd prerequisites mentioned in this reference: https://aws.amazon.com/blogs/security/extend-aws-iam-roles-to-workloads-outside-of-aws-with-iam-roles-anywhere/

An end-entity certificate and associated private key available on the on-premises server

[sxhmilyoyo]

After I install all Amazon Root CAs into the $JAVA_HOME/lib/security/cacerts, the issue is resolved.

Command

keytool -import -trustcacerts -alias AmazonRootCA1 -file /path/to/AmazonRootCA1.pem

Thanks for everyone help, especially @13ajay, your suggestion is correct.