omics service error: could not resolve host

[joseale2310]

Hi,

I am trying to use the omics service using the paws R package. When I try to use anything from the service I get:

omics <- paws::omics(
  config = list(
    credentials = list(
      profile = "saml"
    ), region = "eu-west-1"
  )
)
omics$list_workflows()

Error in curl::curl_fetch_memory(url, handle = handle) : 
  Could not resolve host: omics.eu-west-1.amazonaws.com

I see the omics service is split into workflows, analytics and storage, and I can successfully access my workflows using the AWS CLI. I can also confirm that, for example, s3 works fine using paws::s3(). I guess it is because s3.eu-west-1.amazonaws.com is an actual endpoint in AWS (you can type that in the browser and it will redirect you to https://aws.amazon.com/s3/).

Do I need to provide some different endpoint? Trying to do the following also returns an error:

omics <- paws::omics(
  config = list(
    credentials = list(
      profile = "saml"
    ), region = "eu-west-1",
    endpoint = "workflows-omics.eu-west-1.amazonaws.com"
  )
)
omics$list_workflows()

Error in issue(request$http_request) : no url provided

Session Info

R version 4.2.3 (2023-03-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.6 LTS

Matrix products: default
BLAS:   /opt/R/R-4.2.3/lib/R/lib/libRblas.so
LAPACK: /opt/R/R-4.2.3/lib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_GB.UTF-8       LC_NUMERIC=C               LC_TIME=en_GB.UTF-8        LC_COLLATE=en_GB.UTF-8    
 [5] LC_MONETARY=en_GB.UTF-8    LC_MESSAGES=en_GB.UTF-8    LC_PAPER=en_GB.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] paws_0.6.0

loaded via a namespace (and not attached):
 [1] httr_1.4.7                         compiler_4.2.3                     R6_2.5.1                          
 [4] paws.application.integration_0.6.0 paws.storage_0.6.0                 tools_4.2.3                       
 [7] rstudioapi_0.15.0                  curl_5.2.0                         crayon_1.5.2                      
[10] Rcpp_1.0.12                        paws.common_0.7.3  

[DyfanJones]

Hi sorry about that. I am able to replicate the issue. Not sure about what is causing this bug but will look into it :)

[DyfanJones]

The endpoint (omics.eu-west-1.amazonaws.com) looks correct https://docs.aws.amazon.com/general/latest/gr/healthomics-quotas.html. Will need to double check why it isn't being resolved correctly

[joseale2310]

@DyfanJones thanks! I really appreciate that. I do not know at all what is going on there, I hope you can fix it! I really like the package.

[lyschoening]

@DyfanJones The quoted documentation page is incorrect. HealthOmics is using multiple service endpoints: workflows-omics, tags-omics, storage-omics, control-storage-omics, analytics-omics. See AWS HealthOmics and interface VPC endpoints (AWS PrivateLink).

[joseale2310]

@lyschoening I saw that too! But I do not know then how to provide the paws::omics() the correct endpoint. All my tries have given connection errors.

[DyfanJones]

I think I have found the issue (@lyschoening thanks for highlighting the issue with aws documentation). I believe the issue is coming down to hostPrefix not being picked up in make.paws.

Note: make.paws builds paws sdk from the json apis from aws-sdk-js.

Botocore has takes this into account hostPrefix

https://github.com/boto/botocore/blob/786396c9b236671cc57f6404d84c381ad1499cc5/botocore/serialize.py#L173-L203

[DyfanJones]

As this issue and issue in make.paws it is alot slower to release compared to paws.common (the backbone of the paws sdk). @joseale2310 are you able to test the fixes (once I get to them) paws' r-universe dev packages?

[DyfanJones]

I can update this ticket once I have developed a possible solution

[DyfanJones]

This is to confirm the hostPrefix is missing from the endpoint:

library(paws.storage)

client = omics(config(credentials(profile = "paws"), endpoint = "https://workflows-omics.us-east-1.amazonaws.com"))

client$list_workflows()
#> $items
#> list()
#> 
#> $nextToken
#> character(0)

^{Created on 2024-07-16 with reprex v2.1.1}

[DyfanJones]

Updated make.paws package to take into account host_prefix. Will have to regen the paws sdk before you guys can fully test it out on your own systems :)

[joseale2310]

Whenever you tell me, I will try again. Just tell me what to do :)

[DyfanJones]

I have regenerated paws on r-universe:

install.packages('paws', repos = c('https://paws-r.r-universe.dev', 'https://cloud.r-project.org'))

library(paws.storage)

client = omics(config(credentials(profile = "paws")))

client$list_workflows()

Please have a go and tell me if this fully fixes the issue you have raised 😄 @lyschoening @joseale2310

[DyfanJones]

Note: this should raise your paws packages (paws.storage etc...) to 0.7.0 and paws.common to 0.7.5

[joseale2310]

Hi!

I have reinstalled the packages as per your previous post, restarted my R session and tried to run the last two lines of code. I get this error now:

> client = omics(config(credentials(profile = "saml"), region = "eu-west-1"))
> client$list_workflows()
Error in unclass(e1) + unclass(e2) : 
  non-numeric argument to binary operator

I can confirm that my paws packages are 0.7.0 and paws.common is 0.7.5

[DyfanJones]

Interesting I am not sure why this is appearing. I haven't modified any classes. Are you able to provide a trace to help me debug?

[joseale2310]

This is what i get from traceback. Something to do with date/time?

traceback() 20: .POSIXct(unclass(e1) + unclass(e2), check_tzones(e1, e2)) 19: +.POSIXt(ISOdatetime(1970, 1, 1, 0, 0, 0, tz = "GMT"), sec) 18: convert(node) 17: parse_fn(node, interface) 16: json_parse(node[[node_name]], interface[[name]]) 15: parse_fn(node, interface) 14: json_parse(x, interface[[1]]) 13: FUN(X[[i]], ...) 12: lapply(node, function(x) json_parse(x, interface[[1]])) 11: parse_fn(node, interface) 10: json_parse(node[[node_name]], interface[[name]]) 9: parse_fn(node, interface) 8: json_parse(data, request$data) 7: tag_del(json_parse(data, request$data)) 6: jsonrpc_unmarshal(request) 5: handler$fn(request) 4: run(request, unmarshal) 3: unmarshal(request) 2: send_request(request) 1: client$list_workflows()

[DyfanJones]

oh interesting, that part hasn't been modified 🤔 This might be a new bug I am not aware of. I might need to create a workflow to see why this is causing an error :)

[joseale2310]

At least it changed from one error to another. Progress! On the mean time, I am running a custom function that calls the AWS CLI, writes the output in a json file then loads the json file in R.

[DyfanJones]

@joseale2310 can you try the dev version:

remotes::install_github("dyfanjones/paws/paws.common", ref = "unix_time")

I believe this should fix you issue

[joseale2310]

That did seem to work! Now I do get the API response fine. Thanks!

[DyfanJones]

Perfect, this is going to be a fairly big release for paws. Are you ok using r-universe version for the time being?

[joseale2310]

I will wait until the new release. The workaround is working fine for me so far!