Closed magnussv closed 3 years ago
Hi @magnussv, really sorry to here that you are having issues with RAthena
and noctua
. This is the first time i have come across this type of error. But from a quick google it looks similar to this: https://github.com/ankane/blazer/issues/234 . I will have a look around to see if there is simple solution :)
Thanks @DyfanJones ! I found ankane/blazer#234 when I googled as well, so I tried to run many (new/different) queries but with the same type of result on all of them. I didn't fully understand that issue but the title said "does not allow you to run same query multiple times". :-)
Hi @magnussv, i just want to check some details with you. Does mtcars exist in 2 schema's? For example default.mtcars
and adhoc.mtcars
?
Plus can you let me know how you connect to athena? I am wondering if there is a problem with the defualt schema. I.e
You connect to the default, but you are querying the adhoc schema for example.
con <- dbConnect(noctua:: athena())`
dbGetQuery(con, "select * from adhoc.mtcars")
This looks like a possible solution: https://paws-r.github.io/docs/athena/start_query_execution.html.
The parameter ClientRequestToken
:
A unique case-sensitive string used to ensure the request to create the query is idempotent (executes only once). If another StartQueryExecution request is received, the same response is returned and another query is not created. If a parameter has changed, for example, the QueryString, an error is returned.This token is listed as not required because AWS SDKs (for example the AWS SDK for Java) auto-generate the token for users. If you are not using the AWS SDK or the AWS CLI, you must provide this token or the action will fail.
Could be the issue. I will need to read some boto3, documentation in how to handle this(any suggestions welcome).
In the meantime can you try caching: https://dyfanjones.github.io/noctua/articles/aws_athena_query_caching.html and let me know if this persists.
(Sorry for the mountain of questions please take your time to answer them :) )
From investigating further it looks like there are 2 possible issues going on here.
Error: InvalidRequestException (HTTP 400). Idempotent parameters do not match
: seems to be an issue with start_query_execution
function. The authors of paws
have done a great job in integrating R with AWS and usually the parameter ClientRequestToken
doesn't need to be provided however for what ever reason it is throwing this error. I believe using a uuid string for the ClientRequestToken
should help to fix this.Error: InvalidRequestException (HTTP 400). Could not find results
: is caused when Athena is looking for the result of a repeat query. If that query has previously been removed from s3 then Athena throws an error. This can be replicated with forcing the ClientRequestToken
as digest::digest(paste(statement, conn@info$dbms.name, sep="/"), algo = "md5")
(replicating https://github.com/ankane/blazer/issues/234 solution). By using digest::digest(paste(statement, conn@info$dbms.name, sep="/"), algo = "md5")
the QueryExecutionId
is always the same for repeat queries. As noctua
tries tidy up after it's self (when caching is turned off) the data of past queries is deleted. I believe using uuid string for ClientRequestToken
should resolve this as well.Hi @magnussv can please try out the development branch "start_query_execution"
remotes::install_github("dyfanjones/noctua", ref = "start_query_execution")
Please let me know if the error continues to persist.
@DyfanJones thanks for your quick response. You find answers to your initial questions below.
Hi @magnussv, i just want to check some details with you. Does mtcars exist in 2 schema's? For example
default.mtcars
andadhoc.mtcars
?
mtcars
exist only in adhoc
. The default
is empty. It seems like it doesn't matter which table I'm querying, I get the same issues independent of the table (even with tables with unique names across all databases).
Plus can you let me know how you connect to athena? I am wondering if there is a problem with the defualt schema. I.e
You connect to the default, but you are querying the adhoc schema for example.
con <- dbConnect(noctua:: athena())` dbGetQuery(con, "select * from adhoc.mtcars")
I'm using credentials from a local ~/.aws/credentials with [default]
profile; I have tried with others as well. I have also tried to connect to a specific schema as well, with the same result.
In the meantime can you try caching: https://dyfanjones.github.io/noctua/articles/aws_athena_query_caching.html and let me know if this persists.
I have tried with noctua_options(cache_size = 0)
, noctua_options(cache_size = 10)
, noctua_options(cache_size = 99)
(note: = 100 gives an error), noctua_options()
and noctua_options(clear_cache = T)
but the problem persists.
Hi @magnussv can please try out the development branch "start_query_execution"
remotes::install_github("dyfanjones/noctua", ref = "start_query_execution")
Please let me know if the error continues to persist.
I will try it and come back to you!
Hi @magnussv can please try out the development branch "start_query_execution"
remotes::install_github("dyfanjones/noctua", ref = "start_query_execution")
Please let me know if the error continues to persist.
@DyfanJones I can confirm that this resolved the issues! I ran around 100 queries and all returned the expected result. Great! Thanks for the instant correction. :-) 👍
@magnussv oh brilliant. I didn't know what to do next if that failed 😆 I will merge the branch after I have tweaked the caching option to allow 100 distinct queries (thanks for point that out to me :D )
noctua_1.8.1 has been submitted to the cran.
Hello Dyfan, Lately I had problems with
RAthena
so I turned tonoctua
instead and I noticed some peculiar, almost random, issues.Issue Description
When running the code below then I usually get
Error: InvalidRequestException (HTTP 400). Idempotent parameters do not match
or sometimesError: InvalidRequestException (HTTP 400). Could not find results
HOWEVER about 1 of 5 times I retrieve the desired result. It almost seems random. Issue occurs in both R Console and in Rstudio. As a side note: If I for example uselibrary(aws.s3); bucketlist()
then I always retrieves the desired result, so I know that I'm assuming the role correctly.Reproducible Example
Example:
Screenshot:
Traceback: