search

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.57k stars 3.88k forks source link

(stepfunctions-tasks): Regression in AthenaStartQueryExecution #11832

Open aidansteele opened 3 years ago

aidansteele commented 3 years ago

11203 introduced a small regression wherein you can no longer use a JSON Path to specify properties for a StartQueryExecution AWS service call.

Specifically, outputLocation was changed from a string to this:

https://github.com/aws/aws-cdk/blob/f26a592e609674d528990aad14fb8884112ad64d/packages/%40aws-cdk/aws-stepfunctions-tasks/lib/athena/start-query-execution.ts#L239

Which means that this example in the README no longer compiles - and the functionality it represents does not appear to be possible any longer.

https://github.com/aws/aws-cdk/blob/f26a592e609674d528990aad14fb8884112ad64d/packages/%40aws-cdk/aws-stepfunctions-tasks/README.md#L235

shivlaks commented 3 years ago

@aidansteele - thanks for filing the bug. that is indeed a regression on the ability to use paths. perhaps we will need to offer an OutputOptions type where either an s3 location or a path can be provided (since we can't use type unions) which would enable the more robust structure of an s3 location or a string that we can validate to be a JSON path.

the Athena integration is experimental so it'll be a breaking change we can make. Did you have another idea/thought here?

aidansteele commented 3 years ago

Hi @shivlaks, thanks for such a quick response! Unfortunately I'm still very new to the CDK and Typescript so I can't yet provide any helpful ideas on a solution. I'll try think about it and ask some of my more CDK-experienced colleagues if they want to weigh in. In any case, I think breaking changes are fine because it's experimental, like you said.

LeonLuttenberger commented 2 years ago

Hey, is there any update on this? Our team is also running into this issue in CDK.

Can you recommend also recommend a workaround in the meantime? We are using CDK to generate our step functions, and AthenaStartQueryExecution is just one step in our state machine, but we need the output location to depend on the input.

cad commented 1 year ago

It just blew my mind how this issue is still not fixed after 2 years of being reported. This is a major problem that hinders productive use of step functions with CDK.

Btw @LeonLuttenberger were you or your team able to find a workaround with this?