search

aws / aws-sdk-java-v2

The official AWS SDK for Java - Version 2
Apache License 2.0
2.18k stars 841 forks source link

Default section with region required in .aws/config #1512

Closed Weltraumschaf closed 4 years ago

Weltraumschaf commented 4 years ago

I use some profiles with a ProfileCredentialsProvider to do the SDK operations and my config file has no [default] section.

Expected Behavior

The region information should be used from the given profile.

Current Behavior

The SDK throws the error:

software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@11d045b4: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@17e9bc9e: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@9301672: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@163042ea: Unable to contact EC2 metadata service.]

    at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)
    at software.amazon.awssdk.regions.providers.AwsRegionProviderChain.getRegion(AwsRegionProviderChain.java:70)
    at software.amazon.awssdk.regions.providers.LazyAwsRegionProvider.getRegion(LazyAwsRegionProvider.java:45)
    at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.regionFromDefaultProvider(AwsDefaultClientBuilder.java:178)
    at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.resolveRegion(AwsDefaultClientBuilder.java:167)
    at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.lambda$mergeChildDefaults$0(AwsDefaultClientBuilder.java:111)
    at software.amazon.awssdk.utils.builder.SdkBuilder.applyMutation(SdkBuilder.java:61)
    at software.amazon.awssdk.core.client.config.SdkClientConfiguration.merge(SdkClientConfiguration.java:66)
    at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.mergeChildDefaults(AwsDefaultClientBuilder.java:111)
    at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.syncClientConfiguration(SdkDefaultClientBuilder.java:143)
    at software.amazon.awssdk.services.ec2.DefaultEc2ClientBuilder.buildClient(DefaultEc2ClientBuilder.java:27)
    at software.amazon.awssdk.services.ec2.DefaultEc2ClientBuilder.buildClient(DefaultEc2ClientBuilder.java:22)
    at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:119)
    at de.weltraumschaf.aws.DefaultAwsClientFactory.createEc2Client(DefaultAwsClientFactory.java:52)
    at de.weltraumschaf.FooTest.createVirtualPrivateCloud(FooTest.java:61)
    at de.porsche.zerog.servicebroker.broker.FooTest.foo(FooTest.java:51)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

Steps to Reproduce (for bugs)

Remove the [default] section from the files in ~/.aws and only add the region in the particular profiles.

Your Environment

klaytaybai commented 4 years ago

Hi @Weltraumschaf, can you please add sample code to show how you are specifying the profile you expect to be used?

Weltraumschaf commented 4 years ago

The .aws/config looks like:

[profile snafu]
region = eu-central-1
output = json
role_arn = arn:aws:iam::2...5:role/snafu-01
; source_profile = snafu
role_session_name = Fubar

The .aws/credentials looks like:

[snafu]
aws_access_key_id = A...K
aws_secret_access_key = j...v

I also fiddling around with the source_profile setting in the profile. I'm not sure if it has something to do with this issue. When I add it, then AWS SDK complains about "Invalid profile file: Circular relationship detected with profiles [snafu].". But aws sts get-caller-identity --profile snafu works fine. If I remove source_profile then AWS SDK seems to work, but aws sts ... complains about "Partial credentials found in assume-role, missing: source_profile or credential_source". Fun fact: eksctl works with both w/o any problem. I'm confused about profiles and the documentation I found doesn't solve my problems.

Weltraumschaf commented 4 years ago

Hm, ok... Maybe I should look into Azure or GCP instead of AWS ;-)

debora-ito commented 4 years ago

Hey @Weltraumschaf sorry for the long delay in response here. I'll investigate.

Quick question first: if you have a region in [default] and a different region under the profile, is the region in the profile being used?

Weltraumschaf commented 4 years ago

Yes it is used.

millems commented 4 years ago

The SDK only uses the provided credentials provider to determine the credentials, not the region.

To use the region from a specific profile, you'll have to use the "aws.profile" system property or "AWS_PROFILE" environment variable to specify the profile name.

Alternatively, you can load the region from the profile file manually:

Optional<Region> profileRegion = 
        ProfileFile.defaultProfileFile()
                   .profile("snafu")
                   .map(p -> p.properties().get(ProfileProperty.REGION))
                   .map(Region::of)

We can take a feature request to make this easier to use (we have an internal "region provider" concept we could externalize, for example), since this seems like a confusing experience.

debora-ito commented 4 years ago

I'll go ahead and close this. Feel free to reopen if you have further questions.

Weltraumschaf commented 4 years ago

I load the Profile w/:

ProfileCredentialsProvider.builder()
        .profileName("snafu")
        .build();

There is no good reason why this does not respect the configured region property, if all other tools (awscli and eksctl) does. Also you do not mention that anywhere in the documentation about profiles.

millems commented 4 years ago

You're configuring the client to use the profile file for credentials, not region. In effect this is what you're doing:

S3Client client =
    S3Client.builder()
            .credentialsProvider(ProfileCredentialsProvider.builder().profileName("snafu").build())
            .region(null) // use default region resolution logic
            .build();

It's debatable whether that's a "good reason" but it's the reason. The AWS CLI does not have a way to configure credential providers and region providers separately, so that's not a great example of the Java SDK behaving different than other tools.

If you want behavior equivalent to the AWS CLI, you'll need to use the AWS_PROFILE environment variable or the aws.profile system property:

System.setProperty("aws.profile", "snafu");
S3Client client = S3Client.create();

It's not ideal, but if you want equivalent behavior, that's the least amount of code to do it.

Weltraumschaf commented 4 years ago

Please add this to the JavaDoc of the API!

millems commented 4 years ago

Where would the ideal location be for this documentation that a user would find it?

Weltraumschaf commented 4 years ago

I would prefer a package level documentation with example code. Take a look at the Mockito Framework. This is for me the "gold standard" for good and useful JavaDoc and error/exception messages.

No offense, but the most of the SDK Javadoc is superfluous. Example: That setFoo(foo) has a JavaDoc like "Sets the parameter foo" is not very helpful and it's waste of characters ;-)

IMHO a package level documentation with short examples in an FAQ style like: You want to set a custom region example code like Mockito. That would have the benefit that you do not need to maintain example at a different place.

But at least it should be mentioned at ProfileCredentialsProvider.builder() docuemtation that this defaults to null.

More I think about it: Why does it defaults to null? I try to not use null values or fail early with a propper error message to avoid problems like this. But only my clean coders opinion ;-)

azlconsulting commented 4 years ago

Where would the ideal location be for this documentation that a user would find it?

IMO, this needs to be updated in three places:

  1. https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/java-dg-region-selection.html#region-selection-query-service to:
    • add the 'aws.region' system property as the second potential source
    • make it clear that only the region property of the 'default' profile will be used, even if there is only one profile in the file, unless the AWS_PROFILE environment variable or the aws.profile system property is set to the desired profile.
  2. https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/awscore/client/builder/AwsClientBuilder.html#region-software.amazon.awssdk.regions.Region- to:
    • include the ability to set the config file and profile as per the (updated) previous page
    • state that setting the region is independent of any config file or profile name used in instantiating the credentialsProvider configured for the Builder.
  3. https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/credentials/ProfileCredentialsProvider.Builder.html to:
    • state that any region property set in the profile file or profile name defined for the credentials provider will not be used by the AwsClientBuilder to which the credentials provider is passed unless independently set by the region method.
azlconsulting commented 4 years ago

We can take a feature request to make this easier to use (we have an internal "region provider" concept we could externalize, for example), since this seems like a confusing experience.

Absolutely there should be an easy way to draw the region from the desired profile configuration without having to manually parse the profile file or set system properties or environmental variables.

millems commented 4 years ago

@Weltraumschaf @azlconsulting We have added a defaultProfileName and defaultProfileFile property to the ClientOverrideConfiguration so that you can specify which profile is used for every piece of configuration in the client, regardless of what it is (as a final default, it's still lower priority than other ways of doing it). This includes the region.

We still need to update the docs with this, but feel free to use it before then!

Edit: @azlconsulting I've forwarded your great documentation feedback to our doc writer.

azlconsulting commented 4 years ago

Thanks, @millems. However, it still seems to me (as @Weltraumschaf also appears to have expected) that, if you've used a ProfileCredentialsProvider, any region in the same profile should be passed to the AwsClientBuilder without needing to set defaults or parse the config file manually.

millems commented 4 years ago

@azlconsulting We'll definitely update the documentation, but it sounds like you should then be using the defaultProfileName method mentioned above, not the credentials provider.

The credentials provider configuration only affects the credentials, where the client override configuration affects the whole client, credentials and region included.

azlconsulting commented 4 years ago

@millems, this is the first time I'm doing anything with AWS services and I'm not a developer in the first place but this seems even more confusing to me than the previous situation - that there should be default profile file and name options in the ClientOverrideConfiguration.

Please clarify what you meant by "as a final default, it's still lower priority than other ways of doing it".