ghdna / athena-express

Athena-Express can simplify executing SQL queries in Amazon Athena AND fetching cleaned-up JSON results in the same synchronous or asynchronous request - well suited for web applications.
https://www.npmjs.com/package/athena-express
MIT License
181 stars 70 forks source link

Allow setting ClientRequestToken when starting an query execution #37

Closed jrbarnard closed 4 years ago

jrbarnard commented 4 years ago

If the service running the query times out before the query finishes running / gets off the queue, if it retries it will re start the query. Ideally we'd be able to set the ClientRequestToken through athena express to make an idempotent request.

Suggested syntax:

await athenaExpress.query({
    sql: '...',
    clientRequestToken: '...'
});

I'm happy to PR this, cheers

ghdna commented 4 years ago

Not sure if this is a valid scenario as athena-express waits until the query is finished executing and it does so by checking every 200 milliseconds (default value) on the query status. Furthermore, if you don't care about fetching the results immediately, you can set skipResults: true and use the returned values of QueryExecutionId & S3Location to fetch results later on demand

jrbarnard commented 4 years ago

The use case would be if your request / execution times out (Lambda timeout / api gateway 30 second max request time) before it's exited the Athena express query method. If you then retry the same request it will start a new query rather than be able to pick up where it left off.

Athena's max query time is 30 minutes by default and depending on the time of day multiple minute queue / query times are realistic.

ghdna commented 4 years ago

oh ok. Let me look into it.

ghdna commented 4 years ago

This can now be achieved by setting waitForResults: false in the athena-express configuration. This will return a QueryExecutionId as soon as the query begins. You can then pass QueryExecutionId back into athena-express some time later.