aws-cdk-lib/aws-stepfunctions-tasks: GlueStartJobRun unsupported WAIT_FOR_TASK_TOKEN integration pattern

CodingWithDrew commented 2 months ago

Describe the bug

I am using aws-cdk-lib/aws-stepfunctions-tasks's GlueStartJobRun to create my glue job steps for my step function. These glue jobs execute within ~15-35 seconds, which became an issue with implementing step funcitons as step functions polls glue job executions every minute to check if they completed, increasing our workflow execution time by 2x.

I am trying to use sfn.IntegrationPattern.WAIT_FOR_TASK_TOKEN so my Glue Job can return immediately when it succeeds to step functions to initiate the next step, but the GlueStartJobRun mars this as an unsupported pattern.

Error: Unsupported service integration pattern. Supported Patterns: REQUEST_RESPONSE,RUN_JOB. Received: WAIT_FOR_TASK_TOKEN

I have seen WAIT_FOR_TASK_TOKEN implemented directly through JSON, but not through GlueStartJobRun. What reason is this unsupported?

Regression Issue

[ ] Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

GlueStartJobRun should accept sfn.IntegrationPattern.WAIT_FOR_TASK_TOKEN as an integrationPattern

Current Behavior

Errors with above

Reproduction Steps

Create a GlueStartJobRun with sfn.IntegrationPattern.WAIT_FOR_TASK_TOKEN as the integrationPattern

Possible Solution

Append .waitForTaskToken where you would normally append .sync

Additional Information/Context

No response

CDK CLI Version

4.x

Framework Version

No response

Node.js Version

default.587774.0

OS

Mac

Language

TypeScript

Language Version

No response

Other information

No response

mazyu36 commented 2 months ago

Hi.

I think Glue doesn't support the "wait task for token" feature. Is this correct? https://docs.aws.amazon.com/step-functions/latest/dg/integrate-optimized.html

For this reason, it seems that GlueStartJobRun currently doesn't include WAIT_FOR_TASK_TOKEN. https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-stepfunctions-tasks/lib/glue/start-job-run.ts#L85

CodingWithDrew commented 2 months ago

For Optimized integrations in Step Functions this is not included, but within AWS SDK integrations in Step Functions it is, which leads to the question of why are we limiting the capabilities of this API? I have seen documentation used by others that implements glue with waitForTaskToken using SDK, but this requires writing the entire JSON file rather than creating through CDK.

mazyu36 commented 2 months ago

I think the GlueStartJobRun class is an optimized integration. If you want to use the SDK Integration, I think you need to use the CallAwsService class.

khushail commented 2 months ago

@CodingWithDrew thanks for reporting this.

Thanks @mazyu36 for sharing your insights. Your observations seem to be spot on. The intergration pattern in GlueJobStartRun does not support WAIT_FOR_TOKEN

https://github.com/aws/aws-cdk/blob/76e83a53ec9e102a9aaad629543e0fd84f98faf9/packages/aws-cdk-lib/aws-stepfunctions-tasks/lib/glue/start-job-run.ts#L85

export class GlueStartJobRun extends sfn.TaskStateBase {
  private static readonly SUPPORTED_INTEGRATION_PATTERNS: sfn.IntegrationPattern[] = [
    sfn.IntegrationPattern.REQUEST_RESPONSE,
    sfn.IntegrationPattern.RUN_JOB,
  ];

However in stepfunctions, this is supported -

https://github.com/aws/aws-cdk/blob/d77a1b2f64994bc2e33382b580e0876921b154dc/packages/aws-cdk-lib/aws-stepfunctions/lib/states/task-base.ts#L391

I think this issue should be classified as a Feature request than a bug. Marking it as appropriate and prioritizing it as P2 as it won't be immediately addressed by the team but should be on team's radar. Anyone from the community or team should be able to work on it.

aws / aws-cdk