search

ghdna / athena-express

Athena-Express can simplify executing SQL queries in Amazon Athena AND fetching cleaned-up JSON results in the same synchronous or asynchronous request - well suited for web applications.
https://www.npmjs.com/package/athena-express
MIT License
181 stars 70 forks source link

Skip Fetch Result #14

Closed phasath closed 5 years ago

phasath commented 5 years ago

There should be an option to skip fetch results. I have many cases where I want Athena to process something, save it to S3, return only the QueryExecutionID, and then, download it separately as a csv.

Currently, as far as I know, there is no way to do this.

ghdna commented 5 years ago

So basically download csv at a later time outside of athena-express? There is a way to achieve this. set getStats as true in the athenaExpressConfig and execute your query. This will return the QueryExecutionID that you can capture and use later. Just ignore the results.

phasath commented 5 years ago

I know that I can just ignore, but it will take a while to fetch all my S3 results, and I can't spend workload on this. A flag to skip that step would be much appreciated.

ghdna commented 5 years ago

Executing and waiting for the query to finish in Athena is where you spend the most time & workload. Fetching from S3 is peanuts comparatively. Let me think about the applicability of this feature for other use cases. Meanwhile feel free to fork it and skip that step.

phasath commented 5 years ago

The applicability would be, for example, processing something on backend and just sending the s3 public link to frontend download.

ghdna commented 5 years ago

hmm, okay I'll add it.