s3$list_buckets() hangs

[yonicd]

This code hangs for some reason in R, aws s3 ls --profile "my-profile" works fine from the terminal

s3 <- paws::s3()
s3$list_buckets()

.Rprofile

Sys.setenv(
  AWS_PROFILE = "my-profile",
  AWS_REGION = 'us-east-1'
)

~/.aws/config

$ cat ~/.aws/config
[profile my-profile]
role_arn=arn:aws:iam::123456789012:role/my-role-name
credential_source=Ec2InstanceMetadata

si

```r ─ Session info ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── setting value version R version 4.2.1 (2022-06-23) os Ubuntu 20.04.4 LTS system x86_64, linux-gnu ui RStudio language (EN) collate en_US.UTF-8 ctype en_US.UTF-8 tz Etc/UTC date 2023-01-06 rstudio 2022.07.2+576 Spotted Wakerobin (server) pandoc 2.19.2 @ /usr/local/bin/pandoc ─ Packages ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── ! package * version date (UTC) lib source P cli 3.1.1 2022-01-20 [?] RSPM (R 4.2.0) P clipr 0.7.1 2020-10-08 [?] RSPM (R 4.2.0) P desc 1.4.2 2022-09-08 [?] RSPM (R 4.2.0) P details 0.3.0 2022-03-27 [?] RSPM (R 4.2.0) P digest 0.6.28 2021-09-23 [?] RSPM (R 4.2.0) P fastmap 1.1.0 2021-01-25 [?] RSPM (R 4.2.0) P htmltools 0.5.4 2022-12-07 [?] RSPM (R 4.2.0) P httr 1.4.4 2022-08-17 [?] RSPM (R 4.2.0) P knitr 1.41 2022-11-18 [?] RSPM (R 4.2.0) P magrittr 2.0.2 2022-01-26 [?] RSPM (R 4.2.0) P paws * 0.1.12 2021-09-03 [?] RSPM (R 4.2.0) P png 0.1-8 2022-11-29 [?] RSPM (R 4.2.0) P R6 2.5.1 2021-08-19 [?] RSPM (R 4.2.0) P renv 0.16.0 2022-09-29 [?] RSPM (R 4.2.0) P rlang 1.0.1 2022-02-03 [?] RSPM (R 4.2.0) P rprojroot 2.0.3 2022-04-02 [?] RSPM (R 4.2.0) P sessioninfo 1.2.2 2021-12-06 [?] RSPM (R 4.2.0) P withr 2.4.3 2021-11-30 [?] RSPM (R 4.2.0) P xfun 0.36 2022-12-21 [?] RSPM (R 4.2.0) P xml2 1.3.3 2021-11-30 [?] RSPM (R 4.2.0) [1] /home/rstudio/trials/324-ETD-202/renv/library/R-4.2/x86_64-pc-linux-gnu [2] /home/rstudio/trials/324-ETD-202/renv/sandbox/R-4.2/x86_64-pc-linux-gnu/25ebdc09 P ── Loaded and on-disk path mismatch. ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── ```

[yonicd]

it seems to have to do with the call to retrieve the metadata token from "http://169.254.169.254/latest/api/token" then getting stuck in the httr verbose call

if i put a breakpoint on https://github.com/paws-r/paws/blob/main/paws.common/R/net.R#L119 and manually break from it the job will resolve.

The two problematic urls are:

http://169.254.169.254/latest/meta-data/iam/security-credentials
http://169.254.169.254/latest/meta-data/iam/security-credentials/rstudio-yoni_IAMRole

The rest are fine

https://sts.amazonaws.com/
https://s3.amazonaws.com/

  r <- with_paws_verbose(
    httr::VERB(
      method,
      url = url,
      config = c(httr::add_headers(.headers=headers), dest),
      body = body,
      timeout
    )
  )

if i turn on options(paws.log_level = 3)

then it just hangs on that call

> svc$list_buckets()
DEBUG [2023-01-08 18:27:52.853]: *    Trying 169.254.169.254:80...
DEBUG [2023-01-08 18:27:52.853]: *  TCP_NODELAY set
DEBUG [2023-01-08 18:27:52.853]: *  Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
INFO [2023-01-08 18:27:52.854]: -> PUT /latest/api/token HTTP/1.1
-> Host: 169.254.169.254
-> User-Agent: libcurl/7.68.0 r-curl/4.3.3 httr/1.4.4
-> Accept-Encoding: deflate, gzip, br
-> Accept: application/json, text/xml, application/xml, */*
-> X-aws-ec2-metadata-token-ttl-seconds: 21600
-> Content-Length: 0
-> 
DEBUG [2023-01-08 18:28:07.913]: *  Closing connection 0
DEBUG [2023-01-08 18:28:07.916]: *  Hostname 169.254.169.254 was found in DNS cache
DEBUG [2023-01-08 18:28:07.916]: *    Trying 169.254.169.254:80...
DEBUG [2023-01-08 18:28:07.916]: *  TCP_NODELAY set
DEBUG [2023-01-08 18:28:07.916]: *  Connected to 169.254.169.254 (169.254.169.254) port 80 (#1)
INFO [2023-01-08 18:28:07.917]: -> GET /latest/meta-data/iam/security-credentials HTTP/1.1
-> Host: 169.254.169.254
-> User-Agent: libcurl/7.68.0 r-curl/4.3.3 httr/1.4.4
-> Accept-Encoding: deflate, gzip, br
-> Accept: application/json, text/xml, application/xml, */*
-> 
DEBUG [2023-01-08 18:28:07.917]: *  Mark bundle as not supporting multiuse
INFO [2023-01-08 18:28:07.917]: <- HTTP/1.1 200 OK
INFO [2023-01-08 18:28:07.917]: <- Content-Type: text/plain
INFO [2023-01-08 18:28:07.917]: <- Accept-Ranges: none
INFO [2023-01-08 18:28:07.917]: <- Last-Modified: Sun, 08 Jan 2023 17:49:13 GMT
INFO [2023-01-08 18:28:07.917]: <- Content-Length: 20
INFO [2023-01-08 18:28:07.917]: <- Date: Sun, 08 Jan 2023 18:28:07 GMT
INFO [2023-01-08 18:28:07.918]: <- Server: EC2ws
INFO [2023-01-08 18:28:07.918]: <- Connection: close
INFO [2023-01-08 18:28:07.918]: <- 
DEBUG [2023-01-08 18:28:07.918]: *  Closing connection 1
DEBUG [2023-01-08 18:28:07.924]: *  Hostname 169.254.169.254 was found in DNS cache
DEBUG [2023-01-08 18:28:07.925]: *    Trying 169.254.169.254:80...
DEBUG [2023-01-08 18:28:07.925]: *  TCP_NODELAY set
DEBUG [2023-01-08 18:28:07.925]: *  Connected to 169.254.169.254 (169.254.169.254) port 80 (#2)
INFO [2023-01-08 18:28:07.925]: -> PUT /latest/api/token HTTP/1.1
-> Host: 169.254.169.254
-> User-Agent: libcurl/7.68.0 r-curl/4.3.3 httr/1.4.4
-> Accept-Encoding: deflate, gzip, br
-> Accept: application/json, text/xml, application/xml, */*
-> X-aws-ec2-metadata-token-ttl-seconds: 21600
-> Content-Length: 0
-> 

[DyfanJones]

Sorry about that, interesting I am currently not sure why it is hanging. I will have to try spinning up an rstudio ec2 to see if i can replicate this issue.

In the meantime if you are able to debug the issue please feel free to raise an PR :)

[DyfanJones]

Hmmm unable to replicate issue with cloudformation stack: https://aws.amazon.com/blogs/machine-learning/implement-rstudio-on-your-aws-environment-and-access-your-data-lake-using-aws-lake-formation-permissions/.

Quick question @yonicd do you have a cloud formation stack I can use to replicate this issue :)

[yonicd]

if i straight curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/rstudio-yoni_IAMRole from the terminal, i get the right response back without hanging

{
  "Code" : "Success",
  "LastUpdated" : "2023-01-09T14:00:12Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "ASIA...",
  "SecretAccessKey" : "K4GBB...",
  "Token" : "IQoJb3...",
  "Expiration" : "2023-01-09T20:15:15Z"
}

[DyfanJones]

@yonicd do you mind running the following chunks and output the results for me? I am wanting to double check the outputs to see if everything is being returned as expected :) (sorry i am currently unable to get an environment to replicate yours).

Breaking get_instance_metadata function into it's parts:

query_path <- "iam/security-credentials/rstudio-yoni_IAMRole"

token_ttl <- '21600'
# Get token timeout for IMDSv2 tokens
token <-  ""
metadata_token_url <-  file.path(
  "http://169.254.169.254/latest/api/token"
)
metadata_token_request <- paws.common:::new_http_request(
  "PUT",
  metadata_token_url,
  timeout = 1,
  header=c("X-aws-ec2-metadata-token-ttl-seconds"= token_ttl)
)

metadata_token_response <- tryCatch(
  {
    paws.common:::issue(metadata_token_request)
  },
  error = function(e) {
    NULL
  }
)

Please output:

# checking if token is being collected
is.null(metadata_token_response)
metadata_token_response$status_code == 200
length(metadata_token_response[["body"]])>0

Check if token is created correctly

if (!is.null(metadata_token_response) && metadata_token_response$status_code == 200) {
  if (length(metadata_token_response[["body"]])>0) {
    token <- rawToChar(metadata_token_response[["body"]])
  }
}

Please output:

token

Checking if hangs due to token:

metadata_url <- file.path(
  "http://169.254.169.254/latest/meta-data",
  query_path
)

if (token!="") {
  metadata_request <- paws.common:::new_http_request(
    "GET",
    metadata_url,
    timeout = 1,
    header = c("X-aws-ec2-metadata-token"= token)
  )
} else {
  metadata_request <- paws.common:::new_http_request("GET", metadata_url, timeout = 1)
}

metadata_response <- tryCatch(
  {
    issue(metadata_request)
  },
  error = function(e) {
    NULL
  }
)

[yonicd]

I just checked terminal curl -s http://169.254.169.254/latest/api/token and this hangs

> query_path <- "iam/security-credentials/rstudio-yoni_IAMRole"
> token_ttl <- '21600'
> # Get token timeout for IMDSv2 tokens
> token <-  ""
> metadata_token_url <-  file.path(
+   "http://169.254.169.254/latest/api/token"
+ )
> metadata_token_request <- paws.common:::new_http_request(
+   "PUT",
+   metadata_token_url,
+   timeout = 1,
+   header=c("X-aws-ec2-metadata-token-ttl-seconds"= token_ttl)
+ )
> metadata_token_response <- tryCatch(
+   {
+     paws.common:::issue(metadata_token_request)
+   },
+   error = function(e) {
+     NULL
+   }
+ )
Called from: eval(expr, p)
Browse[1]> n
debug at /home/rstudio/projects/paws/paws.common/R/net.R#120: r <- with_paws_verbose(httr::VERB(method, url = url, config = c(httr::add_headers(.headers = headers), 
    dest), body = body, timeout))
Browse[2]> c
DEBUG [2023-01-09 14:35:34.603]: *    Trying 169.254.169.254:80...
DEBUG [2023-01-09 14:35:34.603]: *  TCP_NODELAY set
DEBUG [2023-01-09 14:35:34.604]: *  Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
INFO [2023-01-09 14:35:34.604]: -> PUT /latest/api/token HTTP/1.1
-> Host: 169.254.169.254
-> User-Agent: libcurl/7.68.0 r-curl/4.9100 httr/1.4.4
-> Accept-Encoding: deflate, gzip, br
-> Accept: application/json, text/xml, application/xml, */*
-> X-aws-ec2-metadata-token-ttl-seconds: 21600
-> Content-Length: 0
-> 
DEBUG [2023-01-09 14:35:46.711]: *  Closing connection 0
> # checking if token is being collected
> is.null(metadata_token_response)
[1] TRUE
> metadata_token_response$status_code == 200
logical(0)
> length(metadata_token_response[["body"]])>0
[1] FALSE
> if (!is.null(metadata_token_response) && metadata_token_response$status_code == 200) {
+   if (length(metadata_token_response[["body"]])>0) {
+     token <- rawToChar(metadata_token_response[["body"]])
+   }
+ }
> token
[1] ""
> if (token!="") {
+   metadata_request <- paws.common:::new_http_request(
+     "GET",
+     metadata_url,
+     timeout = 1,
+     header = c("X-aws-ec2-metadata-token"= token)
+   )
+ } else {
+   metadata_request <- paws.common:::new_http_request("GET", metadata_url, timeout = 1)
+ }
Error in is.url(url) : object 'metadata_url' not found
> metadata_url <- file.path(
+   "http://169.254.169.254/latest/meta-data",
+   query_path
+ )
> if (token!="") {
+   metadata_request <- paws.common:::new_http_request(
+     "GET",
+     metadata_url,
+     timeout = 1,
+     header = c("X-aws-ec2-metadata-token"= token)
+   )
+ } else {
+   metadata_request <- paws.common:::new_http_request("GET", metadata_url, timeout = 1)
+ }
> metadata_request
$method
[1] "GET"

$url
$scheme
[1] "http"

$opaque
[1] ""

$user
[1] ""

$host
[1] "169.254.169.254"

$path
[1] "/latest/meta-data/iam/security-credentials/rstudio-yoni_IAMRole"

$raw_path
[1] ""

$force_query
[1] FALSE

$raw_query
[1] ""

$fragment
[1] ""

attr(,"class")
[1] "struct"

$proto
[1] "HTTP/1.1"

$proto_major
[1] 1

$proto_minor
[1] 1

$header
list()

$body
NULL

$content_length
[1] 0

$transfer_encoding
list()

$close
[1] FALSE

$host
[1] "169.254.169.254"

$form
list()

$post_form
list()

$multipart_form
list()

$trailer
list()

$remote_addr
[1] ""

$request_uri
[1] ""

$tls
NULL

$cancel
NULL

$timeout
[1] 1

$response
NULL

$ctx
list()

$dest
NULL

attr(,"class")
[1] "struct"
> metadata_response <- tryCatch(
+   {
+     issue(metadata_request)
+   },
+   error = function(e) {
+     NULL
+   }
+ )
> metadata_response
NULL

[yonicd]

this is also hanging

$ TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \
> && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/    
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:32 --:--:--     0^C

[DyfanJones]

Ah I think you missed the metadata_url stage to cause the above error.

metadata_url <- file.path(
  "http://169.254.169.254/latest/meta-data",
  query_path
)

if (token!="") {
  metadata_request <- paws.common:::new_http_request(
    "GET",
    metadata_url,
    timeout = 1,
    header = c("X-aws-ec2-metadata-token"= token)
  )
} else {
  metadata_request <- paws.common:::new_http_request("GET", metadata_url, timeout = 1)
}

metadata_response <- tryCatch(
  {
    issue(metadata_request)
  },
  error = function(e) {
    NULL
  }
)

we should have timeouts set to prevent hanging so will have to double check they are set correctly

[DyfanJones]

Oh sorry @yonicd did you mention

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`

This is hanging?

[yonicd]

yes. that is hanging

I am running in a container on an ec2 instance.

It isnt hanging in the ec2, but it is in the container. I am checking now what could be blocking it in the container.

[yonicd]

i changed the network mode in the docker compose yml to 'host' and it works now. did something change in the underlying code to not let the host send back the response when it is bridge mode?

if i set the ports manually, then it still gets blocked from returning a response.

#network_mode: 'host'
ports:
  - 8787:8787
  - 80:80

[DyfanJones]

@yonicd that is great you have got it working :) . There has been an improvement to the code to allow for IMDSV2 https://github.com/paws-r/paws/pull/552, however it shouldn't of changed the response when it is bridge mode. We still haven't implement ipv6 https://github.com/paws-r/paws/issues/561 as of yet, so there is still room for improvement.

@yonicd if you found a bug in the current implementation please feel free to raise a PR or ticket :) As it would be good to improve the sdk for R users :D

[yonicd]

good to hear. i couldnt find a bug in the code of the package. I just made a workaround by using host network mode.

thanks for all the help!

[DyfanJones]

I will close this for now, but please re-open if it come up again 😄