Add a waiter for Athena query execution to be finished

[alfaproject]

Describe the feature

Athena queries are run in the background and there are libraries like Athena Express (which is abandoned) that basically wait until the query is ready to then either download the results or parse them some other way

I think it would be beneficial to have an official waiter for this use case. The concept of waiters is not new to the SDK so it would be just another one, I suppose.

Use Case

Start an Athena query and wait for it to be completed (not queued and not running) to get the execution result

Proposed Solution

We have our own simple layer on top of the SDK to abstract some things, and this is how we implemented this suggestion:

export const startAthenaQueryExecution = (input: StartQueryExecutionCommandInput) =>
  athenaClient().send(new StartQueryExecutionCommand(input));

export const getAthenaQueryExecution = (input: GetQueryExecutionCommandInput) =>
  athenaClient().send(new GetQueryExecutionCommand(input));

export async function executeAthenaQuery(input: StartQueryExecutionCommandInput): Promise<QueryExecution> {
  const { QueryExecutionId: queryExecutionId } = await startAthenaQueryExecution(input);

  let retryIn = 0;
  let queryExecution: QueryExecution;
  do {
    // The query runs asynchronously so wait a bit before checking the result up to 1s per retry
    if (retryIn < 1000) {
      retryIn += 250;
    }
    await setTimeout(retryIn);

    const getQueryExecutionOutput = await getAthenaQueryExecution({ QueryExecutionId: queryExecutionId });
    queryExecution = getQueryExecutionOutput.QueryExecution!;
  } while (
    queryExecution.Status!.State === QueryExecutionState.QUEUED ||
    queryExecution.Status!.State === QueryExecutionState.RUNNING
  );
  return queryExecution;
}

Other Information

No response

Acknowledgements

• [ ] I may be able to implement this feature request
• [ ] This feature might incur a breaking change

SDK version used

3.485.0

Environment details (OS name and version, etc.)

Lambda

[RanVaknin]

Hi @alfaproject ,

Thanks for reaching out again. Waiters are one of the things that the SDK ecosystem code-generates from each service's API model. If an AWS service has defined an operation to be waitable, they should model it as such, specify the I/O and the SDK will create a concrete class that can be used to wait on said operation.

The problem is that often times even if operations are technically waitable (status of resource goes through stages and can be listened on) services would not define that paginator for it in their model (maybe simply an oversight, or sometimes for concrete reasons like things that might seem like should be waited on, but are not reliably waitable), and the SDK ecosystem wont generate it.

To visualize this I will attach the internal models that SDK v2 holds (v3 holds the same model but in a different more hard to read format, but they hold the same data). You can see that Cloudformation model has waiters defined, but the Athena team did not.

Because the SDK is code generated, and relies heavily on the model, we cannot introduce hand written waiters unless they are defined but not generated (because of a model inconsistency or bug).

I hope this does not come across as dismissive, but the new modus operandi is to avoid customizations on service related things as much as possible because those customization are hard to carry over to the next major version and can create feature parity gaps. We aspire to fix everything that can be fixed upstream with the service itself so that the changes carry across all major versions to come, and also so that all the SDKs and CLIs will benefit from that change.

I have created a feature request (P113121613) with the Athena team asking them to add waiters to their model, but I'm not sure when that will happen. I will move this ticket to the cross SDK repo for further tracking, and so that the ticket will remain discoverable for others that may benefit from your code snippet.

Thanks again for taking the time to reach out and contribute. I hope this doesn't discourage you from raising further issues in the future. All the best, Ran~

[RanVaknin]

I have confirmed that the service team has added this to their backlog. Since no further action can be done by the SDK team, I'm going to go ahead and close this.

Please keep an eye on subsequent SDK release notes for updates on this.

Thanks again for reaching out! Ran~

[github-actions[bot]]

This issue is now closed.

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[alfaproject]

@RanVaknin thank you for your extensive explanation. Is there somewhere where I can follow that progress or it's internal? If it is internal, would it be ok to keep this open so that I get notified later? It's really hard to follow the SDK daily updates otherwise

[RanVaknin]

Hi @alfaproject ,

If you have access to AWS support through the AWS Console you can use this ticket ID P113121613 to request a follow up from an internal support team member.

Our team cannot own communication for service related issues so once we verify that a customer request has made it to the right owner, and get a confirmation about will / wont implement we close the issue.

Sorry for the inconvenience. Ran~