Unable to trigger Standard Step Function via App Sync API

khlling commented 7 months ago

Before opening, please confirm:

[X] I have searched for duplicate or closed issues and discussions.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

React

Amplify APIs

GraphQL API

Amplify Version

v6

Amplify Categories

api

Backend

Amplify CLI

Environment information

``` # Put output below this line Browsers: Chrome: 120.0.6099.216 Safari: 17.1 npmPackages: @babel/plugin-proposal-private-property-in-object: ^7.21.11 => 7.21.11 (7.21.0-placeholder-for-preset-env.2) @emotion/react: ^11.4.1 => 11.11.3 @emotion/styled: ^11.3.0 => 11.11.0 @mui/icons-material: ^5.0.3 => 5.15.4 @mui/material: ^5.0.3 => 5.15.4 @mui/styles: ^5.2.3 => 5.15.4 @mui/x-date-pickers: ^6.10.0 => 6.19.0 @testing-library/jest-dom: ^5.11.4 => 5.17.0 @testing-library/react: ^11.1.0 => 11.2.7 @testing-library/user-event: ^12.1.10 => 12.8.3 @toggled-apps/today-types: ^0.11.40 => 0.11.40 @types/jest: ^27.0.2 => 27.5.2 (29.5.11) @types/mui-datatables: ^4.3.10 => 4.3.12 @types/node: ^16.10.3 => 16.18.71 (20.11.2) @types/react: ^18.2.38 => 18.2.48 @types/react-dom: ^18.2.17 => 18.2.18 @types/react-router-dom: ^5.3.1 => 5.3.3 @uiw/react-markdown-editor: ^5.13.1 => 5.13.1 aws-amplify: ^5.3.6 => 5.3.15 dayjs: ^1.11.9 => 1.11.10 jwt-decode: ^3.1.2 => 3.1.2 mui-datatables: ^4.3.0 => 4.3.0 react: ^17.0.2 => 17.0.2 react-dom: ^17.0.2 => 17.0.2 react-dropzone: ^14.2.3 => 14.2.3 react-hook-form: ^7.49.3 => 7.49.3 react-player: ^2.13.0 => 2.14.1 react-router-dom: ^6.8.2 => 6.21.2 react-scripts: ^5.0.1 => 5.0.1 react-webcam: ^6.0.0 => 6.0.1 typescript: ^4.4.3 => 4.9.5 web-vitals: ^1.0.1 => 1.1.2 yarn-audit-fix: ^10.0.7 => 10.0.7 npmGlobalPackages: @aws-amplify/cli: 12.7.1 aws-es-kibana: 1.0.8 commitizen: 4.2.4 cz: 1.8.2 expo-cli: 5.0.3 express-generator: 4.16.1 firebase-tools: 11.29.1 http-server: 14.1.0 jetifier: 2.0.0 lerna: 4.0.0 local-web-server: 5.1.1 n: 9.0.1 npm: 9.6.3 release-it: 14.12.5 typescript: 4.6.3 yarn: 1.22.19 ```

Describe the bug

My flow is as follows:

App sync API Graphql Mutation called
App sync startExectution on Step function

Note the step function is a STANDARD workflow, not an express workflow due to limited timeout time on the express workflow.

I initially had an express workflow but I switched to a standard workflow. I'm using amplify to define the App Sync Schema and custom/cdk to define the step function.

When I call the mutation nothing happens. The step function does not start. No errors visible. No error response from App Sync

Expected behavior

Calling the mutation triggers the start execution

Reproduction steps

I followed this setup guide for the flow initially:

https://aws.amazon.com/blogs/mobile/integrate-aws-step-functions-with-aws-amplify-using-amplify-custom-resources/

Works great as is. I customised it for my purpose and it was working fine. However, I was running into an issue with the initial implementation - the express workflow timeout being only 5 mins. The App sync schema was kept the same.

As a result, I changed express to standard workflow. I can manually execute the standard step function (from the console) and this works as expected.

But now I can't trigger the step function from the mutation.
I've tried to invoke the state machine from the App Sync console manually, but the step function doesn't trigger either

Code Snippet

// Put your code below this line.
"""
Create a new 'Execution' type that will be returned by our call
to the Step Functions workflow.
"""
type Execution {
  name: String
  status: String
  input: String
  executionArn: String
  startDate: String
  stopDate: String
  output: String
}

"""
Mutation that triggers the synchronous execution of our Step
Functions workflow.
"""
type Mutation {
  executeMediaProcessingStateMachine(
    input: String!
    itemId: String!
  ): Execution @aws_api_key 
}

const START_EXECUTION_REQUEST_TEMPLATE = (stateMachineArn: String) => {
  return `
  {
    "version": "2018-05-29",
    "method": "POST",
    "resourcePath": "/",
    "params": {
      "headers": {
        "content-type": "application/x-amz-json-1.0",
        "x-amz-target":"AWSStepFunctions.StartSyncExecution"
      },
      "body": {
        "stateMachineArn": "${stateMachineArn}",
        "input": "{ \\\"input\\\": \\\"$context.args.input\\\", \\\"itemId\\\": \\\"$context.args.itemId\\\"}"
      }
    }
  }
`;
};

const RESPONSE_TEMPLATE = `
## Raise a GraphQL field error in case of a datasource invocation error
#if($ctx.error)
  $util.error($ctx.error.message, $ctx.error.type)
#end
## if the response status code is not 200, then return an error. Else return the body **
#if($ctx.result.statusCode == 200)
    ## If response is 200, return the body.
  $ctx.result.body
#else
    ## If response is not 200, append the response to error block.
  $ctx.result.body
#end
`;

   // Create a service role for SFN to use
    const serviceRole = new iam.Role(this, "Role", {
      assumedBy: new iam.ServicePrincipal(
        "states." + cdk.Stack.of(this).region + ".amazonaws.com"
      ),
    });

    /* 
    Defines the express SFN workflow resource using the state 
    machine definition as well as the service role defined above.
    */
    const stateMachine = new sfn.StateMachine(this, "SyncStateMachine", {
      definition: transcribeStateMachineDefinition,
      stateMachineType: sfn.StateMachineType.STANDARD,
      role: serviceRole,
    });

    // Grant AppSync HTTP data source rights to execute the SFN workflow
    stateMachine.grant(
      httpdatasource.grantPrincipal,
      "states:StartSyncExecution"
    );

    // Creates an IAM role that can be assumed by the AWS AppSync service
    const appsyncStepFunctionsRole = new iam.Role(
      this,
      "SyncStateMachineRole",
      {
        assumedBy: new iam.ServicePrincipal("appsync.amazonaws.com"),
      }
    );

    // Allows the role we defined above to execute express SFN workflows
    appsyncStepFunctionsRole.addToPolicy(
      new iam.PolicyStatement({
        resources: [stateMachine.stateMachineArn],
        actions: ["states:StartSyncExecution"],
      })
    );

    /*
    Adds a GraphQL resolver to our HTTP data source that defines how 
    GraphQL requests and fetches information from our SFN workflow.
    */
    httpdatasource.createResolver("execute-state-machine", {
      typeName: "Mutation",
      fieldName: "executeMediaProcessingStateMachine",
      requestMappingTemplate: appsync.MappingTemplate.fromString(
        START_EXECUTION_REQUEST_TEMPLATE(stateMachine.stateMachineArn)
      ),
      responseMappingTemplate:
        appsync.MappingTemplate.fromString(RESPONSE_TEMPLATE),
    });

Log output

``` // Put your logs below this line ```

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

dreamorosi commented 7 months ago

The sample uses the StartSyncExecution action (as you can see both in the IAM role & request template) which is not available for Standard workflows (see docs).

If you want to use Standard workflows you'll have to modify the CDK above to use the StartExecution action instead, which will make the call asynchronous. This API call returns only the execution Arn and start date (docs) so you'll also have to modify the API and/or the client to be able to fetch the result of the execution at a later time.

I haven't tested either with the sample above, but two options that come to mind are:

return the execution Arn to the client, then use this to poll for the result via an AppSync Query that wraps the DescribeExecution action (docs)
modify the step function so that it sends an event to EventBridge as a last state, and use the newly released feature that allows EventBridge to serve as a source for AppSync Subscriptions (docs). This way the frontend only needs to subscribe to AppSync (potentially with a filter that uses the execution Arn) to get the result

khlling commented 7 months ago

@dreamorosi Thanks for your detailed comment. That really helped me understand the difference between Standard and Express workflows. Especially why there's a difference in execution.

In terms of converting from StartSyncExecution to StartExecution. I've made the following modifications:

"""
Create a new 'Execution' type that will be returned by our call
to the Step Functions workflow.
"""
type Execution {
  status: String
  executionArn: String
  startDate: String
}

"""
Mutation that triggers the synchronous execution of our Step
Functions workflow.
"""
type Mutation {
  executeMediaProcessingStateMachine(
    input: String!
    itemId: String!
  ): Execution @aws_api_key
}

const START_EXECUTION_REQUEST_TEMPLATE = (stateMachineArn: String) => {
  return `
  {
    "version": "2018-05-29",
    "method": "POST",
    "resourcePath": "/",
    "params": {
      "headers": {
        "content-type": "application/x-amz-json-1.0",
        "x-amz-target":"AWSStepFunctions.StartExecution"
      },
      "body": {
        "stateMachineArn": "${stateMachineArn}",
        "input": "{ \\\"input\\\": \\\"$context.args.input\\\", \\\"itemId\\\": \\\"$context.args.itemId\\\"}"
      }
    }
  }
`;
};

const RESPONSE_TEMPLATE = `
## Raise a GraphQL field error in case of a datasource invocation error
#if($ctx.error)
  $util.error($ctx.error.message, $ctx.error.type)
#end
## if the response status code is not 200, then return an error. Else return the body **
#if($ctx.result.statusCode == 200)
    ## If response is 200, return the body.
  $ctx.result.body
#else
    ## If response is not 200, append the response to error block.
  $ctx.result.statusCode
#end
`;

    // Create a service role for SFN to use
    const serviceRole = new iam.Role(this, "Role", {
      assumedBy: new iam.ServicePrincipal(
        "states." + cdk.Stack.of(this).region + ".amazonaws.com"
      ),
    });

    /* 
    Defines the express SFN workflow resource using the state 
    machine definition as well as the service role defined above.
    */
    const stateMachine = new sfn.StateMachine(this, "StateMachine", {
      definition: transcribeStateMachineDefinition,
      stateMachineType: sfn.StateMachineType.STANDARD,
      role: serviceRole,
    });

    // Grant AppSync HTTP data source rights to execute the SFN workflow
    stateMachine.grant(httpdatasource.grantPrincipal, "states:StartExecution");

    // Creates an IAM role that can be assumed by the AWS AppSync service
    const appsyncStepFunctionsRole = new iam.Role(
      this,
      "SyncStateMachineRole",
      {
        assumedBy: new iam.ServicePrincipal("appsync.amazonaws.com"),
      }
    );

    // Allows the role we defined above to execute express SFN workflows
    appsyncStepFunctionsRole.addToPolicy(
      new iam.PolicyStatement({
        resources: [stateMachine.stateMachineArn],
        actions: ["states:StartExecution"],
      })
    );

    /*
    Adds a GraphQL resolver to our HTTP data source that defines how 
    GraphQL requests and fetches information from our SFN workflow.
    */
    httpdatasource.createResolver("execute-state-machine", {
      typeName: "Mutation",
      fieldName: "executeMediaProcessingStateMachine",
      requestMappingTemplate: appsync.MappingTemplate.fromString(
        START_EXECUTION_REQUEST_TEMPLATE(stateMachine.stateMachineArn)
      ),
      responseMappingTemplate:
        appsync.MappingTemplate.fromString(RESPONSE_TEMPLATE),
    });
  }
}

However, I still can't get the step function to trigger. When I try and trigger it through the App Sync console I get an empty return object:

{
  "data": {
    "executeMediaProcessingStateMachine": {
      "executionArn": null,
      "status": null,
      "startDate": null
    }
  }
}

Am I missing something?

khlling commented 7 months ago

I've figured it out my http datasource was pointing to https://sync-states." + cdk.Stack.of(this).region + ".amazonaws.com

    // Adds the AWS Step Functions (SFN) service endpoint as a new HTTP data source to the GraphQL API
    const httpdatasource = api.addHttpDataSource(
      "ds",
      "https://sync-states." + cdk.Stack.of(this).region + ".amazonaws.com",
      {
        name: "MediaProcessingSFN",
        authorizationConfig: {
          signingRegion: cdk.Stack.of(this).region,
          signingServiceName: "states",
        },
      }
    );

When it should have been https://states." + cdk.Stack.of(this).region + ".amazonaws.com

    // Adds the AWS Step Functions (SFN) service endpoint as a new HTTP data source to the GraphQL API
    const httpdatasource = api.addHttpDataSource(
      "ds",
      "https://states." + cdk.Stack.of(this).region + ".amazonaws.com",
      {
        name: "MediaProcessingSFN",
        authorizationConfig: {
          signingRegion: cdk.Stack.of(this).region,
          signingServiceName: "states",
        },
      }
    );

aws-amplify / amplify-js