STSClient does not use environment variable for region controls

nbbeeken commented 1 year ago

Checkboxes for prior research

[X] I've gone through Developer Guide and API reference
[X] I've checked AWS Forums and StackOverflow.
[X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

When invoking fromNodeProviderChain using IAM AssumeRoleWithWebIdentity I expect environment variables AWS_REGION / AWS_DEFAULT_REGION / AWS_STS_REGIONAL_ENDPOINTS to control the region that the STSClient uses to send the http request. However, it can be observed that the request is always routed to us-east-1 unless the region is programmatically set.

Documentation References:

SDK version number

@aws-sdk/credential-providers@3.391.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.2.0

Reproduction Steps

Running the fromNodeProviderChain function with http debugging turned on, like so:

node="$(which node)"
env -i \
  NODE_DEBUG="http" \
  AWS_WEB_IDENTITY_TOKEN_FILE="..path../web_token_file" \
  AWS_ROLE_ARN="arn:aws:iam::xxxxxxxxxxxx:role/xxx" \
  AWS_STS_REGIONAL_ENDPOINTS="regional" \
  AWS_REGION="us-west-2" \
  $node -e \
  "require('@aws-sdk/credential-providers').fromNodeProviderChain()().then(() => console.log('success'), e => console.log('error', e))"

Observed Behavior

Logs

HTTP 30987: createConnection sts.us-east-1.amazonaws.com:443

to the terminal, indicating the region variable and the sts_regional_endpoints are not having the desired effect.

Expected Behavior

The AWS_REGION variable along with the AWS_STS_REGIONAL_ENDPOINTS setting should have made the API call contact sts.us-west-2.amazonaws.com:443

Possible Solution

No response

Additional Information/Context

No response

yenfryherrerafeliz commented 1 year ago

Hi @nbbeeken, I can confirm that the default STS client, used internally to resolve credentials with web identity token, does not consider the AWS_DEFAULT_REGION variable to populate the region, instead it considers just either a region passed as parameter or the default one which is us-east-1. This can be confirmed in the following implementation here. However, I need to discuss with the team regarding if this is the expected behavior or not. As workaround, you have the option to explicitly provide the region you want the STS client to use, as follow:

import {ListBucketsCommand, S3Client} from "@aws-sdk/client-s3";
import {fromNodeProviderChain} from "@aws-sdk/credential-providers";

const credProvider = fromNodeProviderChain({
    clientConfig: {
        region: "YOUR-DESIRED-STS-REGION"
    }
});
const client = new S3Client({
    region: "us-east-2",
    credentials: credProvider
})
const response = await client.send(new ListBucketsCommand({}));

console.log(response)

Please let me know if that helps!

Thanks!

nbbeeken commented 1 year ago

Thank you for looking into this @yenfryherrerafeliz! As a library author providing direct control over these options means adding more knobs to our public API (ex. a pass-through object for users to provide their own options). It would be nice to avoid the need for bespoke controls like this when the docs point to these environment variables as the proper controls.

Thanks for the code sample, good to know I am on the right track with the workaround! You can see https://github.com/mongodb/node-mongodb-native/pull/3831 we have translated the environment variables to options in the MongoDB driver since we need this to apply to existing versions of the SDK.

gabegorelick commented 1 year ago

This is definitely surprising behavior. At the very least, this needs to be documented. From what I can tell, it's not documented anywhere.

swcm-mnestler commented 1 year ago

I would go a step further and say that this is documented as "supported", and the fact that it does not work is a bug. Please at least update the documentation to reflect the actual behavior

trivikr commented 4 months ago

I verified from internal documentation that STS Regional Endpoints are not required to be supported in major version bump

SDK major version bumps SHOULD ignore the STS Regional Endpoints option, and only resolve STS endpoints to the partition's STS regional endpoints. A major version bump SDK MAY choose to keep the STS Regional Endpoints option, but the option's default value must be regional.

I'll check the behavior in JS SDK v2 vs v3 in subsequent comments.

trivikr commented 4 months ago

Behavior in JS SDK v2

import AWS from "aws-sdk"; // v2.1659.0
const client = new AWS.STS();

const req = client.getCallerIdentity();
await req.promise();

console.log(req.httpRequest.endpoint.href);

us-west-2

The legacy and regional endpoints are different, as required.

$ AWS_REGION=us-west-2 AWS_STS_REGIONAL_ENDPOINTS=legacy node index.v2.mjs
https://sts.amazonaws.com/

$ AWS_REGION=us-west-2 AWS_STS_REGIONAL_ENDPOINTS=regional node index.v2.mjs
https://sts.us-west-2.amazonaws.com/

us-west-2-fips

The legacy and regional endpoints are same, as required.

$ AWS_REGION=us-west-2-fips AWS_STS_REGIONAL_ENDPOINTS=legacy node index.v2.mjs
https://sts-fips.us-west-2.amazonaws.com/

$ AWS_REGION=us-west-2-fips AWS_STS_REGIONAL_ENDPOINTS=regional node index.v2.mjs
https://sts-fips.us-west-2.amazonaws.com/

trivikr commented 4 months ago

Behavior in JS SDK v3

import { STS, GetCallerIdentityCommand } from "@aws-sdk/client-sts"; // v3.614.0
import { getEndpointFromInstructions } from "@smithy/middleware-endpoint";

const client = new STS();
const command = new GetCallerIdentityCommand();

const endpoint = await getEndpointFromInstructions(
  command.input,
  GetCallerIdentityCommand,
  client.config
);

console.log(endpoint.url.href);

us-west-2

Ignores AWS_STS_REGIONAL_ENDPOINTS and returns regional behavior, as required by major version.

$ AWS_REGION=us-west-2 AWS_STS_REGIONAL_ENDPOINTS=legacy node index.v3.mjs
https://sts.us-west-2.amazonaws.com/

$ AWS_REGION=us-west-2 AWS_STS_REGIONAL_ENDPOINTS=regional node index.v3.mjs
https://sts.us-west-2.amazonaws.com/

us-west-2-fips

Ignores AWS_STS_REGIONAL_ENDPOINTS and returns regional behavior, as required by major version.

$ AWS_REGION=us-west-2-fips AWS_STS_REGIONAL_ENDPOINTS=legacy node index.v3.mjs
https://sts-fips.us-west-2.amazonaws.com/

$ AWS_REGION=us-west-2-fips AWS_STS_REGIONAL_ENDPOINTS=regional node index.v3.mjs
https://sts-fips.us-west-2.amazonaws.com/

trivikr commented 4 months ago

I would go a step further and say that this is documented as "supported", and the fact that it does not work is a bug. Please at least update the documentation to reflect the actual behavior

I've created an internal ticket at V1453005105 to update the Developer Guide, and have followed up on the internal ticket. This issue can be resolved once Developer Guide is updated.

nbbeeken commented 4 months ago

Hey @trivikr thanks for taking a look at this! The original issue was focused on the @aws-sdk/credential-providers package and the fromNodeProviderChain API it provides, which isn't mentioned in your code snippets. It still appears as though that API does not respond to the environment variables, is that intentional?

afarah1 commented 2 weeks ago

@trivikr Here is a code snippet showing the behavior change from v2 to v3, it might be worth it to mention in the v2 to v3 differences document:

v3 sample

const { S3Client, ListBucketsCommand } = require("@aws-sdk/client-s3");
const { fromTokenFile } = require("@aws-sdk/credential-providers");

const listBuckets = async () => {
  try {
    const s3Client = new S3Client({
      credentials: fromTokenFile(),
    });
    const data = await s3Client.send(new ListBucketsCommand({}));
    console.log("Success", data.Buckets);
  } catch (err) {
    console.log("Error", err);
  }
};

listBuckets();

v3 behavior: ignores env variables (must use fromTokenFile({region: process.env.AWS_REGION}) for example).

v2 sample:

const AWS = require('aws-sdk');

const credentials = new AWS.TokenFileWebIdentityCredentials();
const s3 = new AWS.S3({ credentials });
s3.listBuckets((err, data) => {
  if (err) {
    console.log('Error', err);
  } else {
    console.log('Bucket List', data.Buckets);
  }
});

v2 behavior: does NOT ignore env vars

aws / aws-sdk-js-v3