KoteiIto / node-athena

a nodejs simple aws athena client
MIT License
105 stars 73 forks source link

CTAS results from a promise #57

Open GeoffThorpeFT opened 4 years ago

GeoffThorpeFT commented 4 years ago

Great library! I've been using it to execute and capture the results of a variety of Athena SQL commands. They all work apart from the CREATE TABLE AS.

When I execute the following I get a NoSuchKey: The specified key does not exist. error.

    const query = 'CREATE TABLE newtable WITH (format='ORC') SELECT * from rawtable';
        athena-client
        .execute(query)
        .toPromise()
        .then(result => {})
                .catch(error => {})
GeoffThorpeFT commented 4 years ago

Further investigation is suggesting that the NoSuchKey is an attempt to read output from the S3 bucket that was defined when the athena-client was configured. Normally that would be possible since the execution of athena commands triggers the output of results into that bucket. However, a CTAS command does not work in the same way - its results are the table it created. That table could be created anywhere. Therefore attempts to use the result.queryExecution.ResultConfiguration.OutputLocation variable wont be successful as the file is not created.

mathoshek commented 4 years ago

Hello @GeoffThorpeFT, i came across the same issue. Is there any way you managed to make it work?

GeoffThorpeFT commented 4 years ago

@mathoshek Sorry I was not able to get it to work directly. Instead I played with a route where I wrote the output to a bucket. I then created an athena external table from that bucket and re-read the data. Unfortunately that doesn't really improve the performance :(