Ambient AssumeRoleWithWebIdentity credentials do not work correctly when `AWS_DEFAULT_REGION` is unset

[atheriel]

Attempting to use ambient credentials furnished by OIDC (e.g. on EKS or in Posit Workbench) does not work correctly without explicitly setting the default region in an environment variable.

To illustrate this, you'd expect that role assumption would work when AWS_ROLE_ARN, AWS_ROLE_SESSION_NAME, and AWS_WEB_IDENTITY_TOKEN_FILE are set:

> grep("AWS_", names(Sys.getenv()), value = TRUE, fixed = TRUE)
[1] "AWS_ROLE_ARN"                "AWS_ROLE_SESSION_NAME"       "AWS_WEB_IDENTITY_TOKEN_FILE"
> paws::sts()$get_caller_identity()
Error in get_region(cfgs[["credentials"]][["profile"]]) : 
  No region provided

For reference, this works fine with the AWS CLI (and every other SDK I know of):

$ env | grep AWS
AWS_ROLE_ARN=<omitted>
AWS_ROLE_SESSION_NAME=<omitted>
AWS_WEB_IDENTITY_TOKEN_FILE=<omitted>
$ aws sts get-caller-identity
{
    "UserId": "AROAZI3I5KIFBR4H5654I:<role-session-name omitted>",
    "Account": "<omitted>",
    "Arn": "arn:aws:sts::<omitted>:assumed-role/<role omitted>/<role-session-name omitted>"
}

Attempting to pass the region as part of the configuration does not work, either:

> paws::sts(config = list(region = "us-west-2"))$get_caller_identity()
Error in get_region(cfgs[["credentials"]][["profile"]]) : 
  No region provided

To make it work you need to set AWS_DEFAULT_REGION or AWS_REGION:

> Sys.setenv("AWS_DEFAULT_REGION" = "us-west-1")
> paws::sts()$get_caller_identity()
$UserId
[1] "AROAZI3I5KIFBR4H5654I:<role-session-name omitted>"

$Account
[1] "<omitted>"

$Arn
[1] "arn:aws:sts::<omitted>:assumed-role/<role omitted>/<role-session-name omitted>"

It seems like paws should be able to skip the region requirement here, as with other SDKs and the CLI.

[DyfanJones]

It is interesting to see this failing:

> paws::sts(config = list(region = "us-west-2"))$get_caller_identity()
Error in get_region(cfgs[["credentials"]][["profile"]]) : 
  No region provided

are you able to do a traceback for me. Also can you include the version of paws.common you are using :)

[DyfanJones]

library(paws)
options(paws.log_level = 3L)
sts(config = list(region = "us-west-2"))$get_caller_identity()
#> INFO [2024-01-02 18:15:14.356]: -> POST / HTTP/1.1
#> -> Host: sts.us-west-2.amazonaws.com
#> -> Accept-Encoding: deflate, gzip
#> -> User-Agent: paws/0.6.4 (R4.3.2; darwin20; aarch64)
#> -> Accept: application/xml
#> -> Content-Type: application/x-www-form-urlencoded; charset=utf-8
#> -> Content-Length: 43
#> -> X-Amz-Date: 20240102T181513Z
#> -> Authorization: AWS4-HMAC-SHA256 Credential=Omitted/20240102/us-west-2/sts/aws4_request, SignedHeaders=accept;content-length;content-type;host;x-amz-date, Signature=Omitted
#> -> 
#> INFO [2024-01-02 18:15:14.356]: >> Action=GetCallerIdentity&Version=2011-06-15
#> 
#> INFO [2024-01-02 18:15:14.550]: <- HTTP/1.1 200 OK
#> INFO [2024-01-02 18:15:14.550]: <- x-amzn-RequestId: bae0b6f0-f0ba-4ec6-9afd-604667c9495a
#> INFO [2024-01-02 18:15:14.550]: <- Content-Type: text/xml
#> INFO [2024-01-02 18:15:14.551]: <- Content-Length: 408
#> INFO [2024-01-02 18:15:14.551]: <- Date: Tue, 02 Jan 2024 18:15:14 GMT
#> INFO [2024-01-02 18:15:14.551]: <- 
#> $UserId
#> [1] "Omitted"
#> 
#> $Account
#> [1] "123456789"
#> 
#> $Arn
#> [1] "arn:aws:iam::123456789:user/Omitted"

sts()$get_caller_identity()
#> INFO [2024-01-02 18:15:14.823]: -> POST / HTTP/1.1
#> -> Host: sts.eu-west-1.amazonaws.com
#> -> Accept-Encoding: deflate, gzip
#> -> User-Agent: paws/0.6.4 (R4.3.2; darwin20; aarch64)
#> -> Accept: application/xml
#> -> Content-Type: application/x-www-form-urlencoded; charset=utf-8
#> -> Content-Length: 43
#> -> X-Amz-Date: 20240102T181514Z
#> -> Authorization: AWS4-HMAC-SHA256 Credential=Omitted/20240102/eu-west-1/sts/aws4_request, SignedHeaders=accept;content-length;content-type;host;x-amz-date, Signature=Omitted
#> -> 
#> INFO [2024-01-02 18:15:14.824]: >> Action=GetCallerIdentity&Version=2011-06-15
#> 
#> INFO [2024-01-02 18:15:14.873]: <- HTTP/1.1 200 OK
#> INFO [2024-01-02 18:15:14.873]: <- x-amzn-RequestId: a1667bd4-0990-44af-980c-1f09d98ce54a
#> INFO [2024-01-02 18:15:14.874]: <- Content-Type: text/xml
#> INFO [2024-01-02 18:15:14.874]: <- Content-Length: 408
#> INFO [2024-01-02 18:15:14.874]: <- Date: Tue, 02 Jan 2024 18:15:14 GMT
#> INFO [2024-01-02 18:15:14.874]: <- 
#> $UserId
#> [1] "Omitted"
#> 
#> $Account
#> [1] "123456789"
#> 
#> $Arn
#> [1] "arn:aws:iam::123456789:user/Omitted"

^{Created on 2024-01-02 with reprex v2.0.2}

Session info

``` r sessioninfo::session_info() #> ─ Session info ─────────────────────────────────────────────────────────────── #> setting value #> version R version 4.3.2 (2023-10-31) #> os macOS Sonoma 14.0 #> system aarch64, darwin20 #> ui X11 #> language (EN) #> collate en_US.UTF-8 #> ctype en_US.UTF-8 #> tz Europe/London #> date 2024-01-02 #> pandoc 3.1.9 @ /opt/homebrew/bin/ (via rmarkdown) #> #> ─ Packages ─────────────────────────────────────────────────────────────────── #> package * version date (UTC) lib source #> cli 3.6.2 2023-12-11 [1] CRAN (R 4.3.1) #> crayon 1.5.2 2022-09-29 [1] CRAN (R 4.3.0) #> curl 5.2.0 2023-12-08 [1] CRAN (R 4.3.1) #> digest 0.6.33 2023-07-07 [1] CRAN (R 4.3.0) #> evaluate 0.23 2023-11-01 [1] CRAN (R 4.3.1) #> fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.3.0) #> fs 1.6.3 2023-07-20 [1] CRAN (R 4.3.0) #> glue 1.6.2 2022-02-24 [1] CRAN (R 4.3.0) #> htmltools 0.5.7 2023-11-03 [1] CRAN (R 4.3.1) #> httr 1.4.7 2023-08-15 [1] CRAN (R 4.3.0) #> knitr 1.45 2023-10-30 [1] CRAN (R 4.3.1) #> lifecycle 1.0.4 2023-11-07 [1] RSPM (R 4.3.0) #> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.3.0) #> paws * 0.4.0 2023-09-15 [1] CRAN (R 4.3.0) #> paws.common 0.6.4 2023-11-11 [1] CRAN (R 4.3.1) #> paws.security.identity 0.4.0 2023-09-11 [1] CRAN (R 4.3.0) #> purrr 1.0.2 2023-08-10 [1] CRAN (R 4.3.0) #> R.cache 0.16.0 2022-07-21 [1] CRAN (R 4.3.0) #> R.methodsS3 1.8.2 2022-06-13 [1] CRAN (R 4.3.0) #> R.oo 1.25.0 2022-06-12 [1] CRAN (R 4.3.0) #> R.utils 2.12.3 2023-11-18 [1] CRAN (R 4.3.1) #> R6 2.5.1 2021-08-19 [1] CRAN (R 4.3.0) #> Rcpp 1.0.11 2023-07-06 [1] CRAN (R 4.3.0) #> reprex 2.0.2 2022-08-17 [1] CRAN (R 4.3.0) #> rlang 1.1.2 2023-11-04 [1] RSPM (R 4.3.0) #> rmarkdown 2.25 2023-09-18 [1] CRAN (R 4.3.1) #> rstudioapi 0.15.0 2023-07-07 [1] CRAN (R 4.3.0) #> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.3.0) #> styler 1.10.2 2023-08-29 [1] CRAN (R 4.3.0) #> vctrs 0.6.5 2023-12-01 [1] CRAN (R 4.3.1) #> withr 2.5.2 2023-10-30 [1] CRAN (R 4.3.1) #> xfun 0.41 2023-11-01 [1] CRAN (R 4.3.1) #> xml2 1.3.6 2023-12-04 [1] CRAN (R 4.3.1) #> yaml 2.3.8 2023-12-11 [1] CRAN (R 4.3.1) #> #> [1] /Users/dyfanjones/Library/R/arm64/4.3/library #> [2] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library #> #> ────────────────────────────────────────────────────────────────────────────── ```

Hmmm interestingly when I include the logs in these two calls, the region is getting changed when hardcoding it.

[atheriel]

are you able to do a traceback for me. Also can you include the version of paws.common you are using :)

Sure!

> options(paws.log_level = 3L)
> paws::sts(config = list(region = "us-west-2"))$get_caller_identity()
INFO [2024-01-02 13:41:51.754]: Unable to locate credentials file
INFO [2024-01-02 13:41:51.754]: Unable to locate config file
Error in get_region(cfgs[["credentials"]][["profile"]]) : 
  No region provided
> traceback()
18: stop("No region provided")
17: get_region(cfgs[["credentials"]][["profile"]])
16: client_config(service_name = metadata$service_name, endpoints = metadata$endpoints, 
        cfgs = cfgs, service_id = metadata$service_id)
15: new_service(.sts$metadata, handlers, config)
14: .sts$service(config)
13: svc$assume_role_with_web_identity(RoleArn = role_arn, RoleSessionName = role_session_name, 
        WebIdentityToken = web_identity_token)
12: get_assume_role_with_web_identity_creds(role_arn = get_role_arn(), 
        role_session_name = get_role_session_name(), web_identity_token = get_web_identity_token())
11: get_container_credentials_eks()
10: f()
9: call_with_args(provider, credentials)
8: get_credentials(signer$credentials)
7: sign_with_body(v4, request$http_request, request$body, name, 
       region, request$expire_time, request$expire_time > 0, signing_time)
6: sign_sdk_request_with_curr_time(request)
5: handler$fn(request)
4: run(request, sign)
3: sign(request)
2: send_request(request)
1: paws::sts(config = list(region = "us-west-2"))$get_caller_identity()
> packageVersion("paws.common")
[1] ‘0.6.4’

[DyfanJones]

Thanks for the traceback, I will have a look to see if I can replicate this issue :)

[DyfanJones]

I think I know what is going on, but will have a deeper look at it tomorrow. I believe I can get this in 0.7.0 paws.cmmon. @atheriel are you available for testing if possible?

[atheriel]

@DyfanJones Absolutely, I have a few environments that look like this for testing.

[DyfanJones]

@atheriel I believe I have a solution, please try out:

remotes::install_github("dyfanjones/paws/paws.common", ref = "aws-global")

[atheriel]

STS seems to be working now!

> Sys.unsetenv("AWS_DEFAULT_REGION")
> paws::sts()$get_caller_identity()
$UserId
[1] "AROAZI3I5KIFBR4H5654I:<role-session-name omitted>"

$Account
[1] "<omitted>"

$Arn
[1] "arn:aws:sts::<omitted>:assumed-role/<role omitted>/<role-session-name omitted>"

However, regional services don't seem to work:

> paws::s3()$list_buckets()
Error in resolver_endpoint(service_name, region, endpoints, sts_regional_endpoint) : 
  No region provided and no global region found.
> traceback()
6: stop("No region provided and no global region found.")
5: resolver_endpoint(service_name, region, endpoints, sts_regional_endpoint)
4: client_config(service_name = metadata$service_name, endpoints = metadata$endpoints, 
       cfgs = cfgs, service_id = metadata$service_id)
3: new_service(.s3$metadata, handlers, config)
2: .s3$service(config)
1: paws::s3()$list_buckets()

Unless you provide region, which also works now:

> paws::s3(config = list(region = "us-west-2"))$list_buckets()
<omitted>

In contrast, aws s3 ls seems to work fine without a region. Does it use some kind of default?

[DyfanJones]

not sure with aws cli. however i do know aws s3 doesn't have a global region. hence it needing a region

[DyfanJones]

Does it work in boto3? If so I might be missing something

[atheriel]

Heh, it looks like this is actually a quirk of the CLI: for S3 specifically it lists buckets in all regions.

So I think paws is working correctly now. :smile:

[DyfanJones]

that is good to hear :) when cran allows submissions after the xmas break, i will release paws.common 0.7.0 :)

[DyfanJones]

Oh hang on it looks like s3 service defaults to us-east-1 when region isn't set within botocore

https://github.com/boto/botocore/blob/046058dbec07e3292a88fa05e1a997573ce48709/botocore/regions.py#L189

[DyfanJones]

@atheriel please have a go. This should fix the s3 issue as well.

remotes::install_github("paws-r/paws/paws.common")

[atheriel]

:tada:

> Sys.unsetenv("AWS_DEFAULT_REGION")
> paws::s3()$list_buckets()$Buckets |> length()
[1] 52

[DyfanJones]

Closing ticket as paws.common 0.7.0 has been released to the cran