aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.38k stars 3.79k forks source link

aws_cdk.aws_stepfunctions_tasks.HttpInvoke: Get API Endpoint from state input #30749

Open bc-venkata-edara opened 1 week ago

bc-venkata-edara commented 1 week ago

Describe the bug

Get API Endpoint from state input is not working throwing this error.

SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is not valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition

Expected Behavior

It should create stepfunction shown in below image with below definition.

Screenshot 2024-07-03 at 8 11 41 PM
"marketo-leads-creation-update": {
      "Type": "Task",
      "Resource": "arn:aws:states:::http:invoke",
      "Parameters": {
        "Authentication": {
          "ConnectionArn": "arn:aws:events:<region>:<account>:connection/<connection>
        },
        "RequestBody.$": "$.detail.object",
        "Method.$": "$.detail.object_method",
        "ApiEndpoint.$": "States.Format('https://<endpoint>/rest/v1/{}.json', $.detail.object_name)"
      }

Current Behavior

When we add code for step function httpinvoke task cdk python. it's throwing below error.

UPDATE_FAILED        | AWS::StepFunctions::StateMachine | EXAMPLERESOURCESsfMarketosync1464C05F
Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is n
ot valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Request ID: 3c
714a80-85d1-4166-9c23-a61ed3749abd; Proxy: null)" (RequestToken: 186ba6c7-912e-b3a1-e6f9-408473474a6b, HandlerErrorCode: InvalidRequest)

 ❌  EIP-Example failed: Error: The stack named EIP-Example failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is not valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Request ID: 3c714a80-85d1-4166-9c23-a61ed3749abd; Proxy: null)" (RequestToken: 186ba6c7-912e-b3a1-e6f9-408473474a6b, HandlerErrorCode: InvalidRequest)
    at FullCloudFormationDeployment.monitorDeployment (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:451:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:199716)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:181438

 ❌ Deployment failed: Error: The stack named EIP-Example failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is not valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Request ID: 3c714a80-85d1-4166-9c23-a61ed3749abd; Proxy: null)" (RequestToken: 186ba6c7-912e-b3a1-e6f9-408473474a6b, HandlerErrorCode: InvalidRequest)
    at FullCloudFormationDeployment.monitorDeployment (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:451:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:199716)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:181438

The stack named EIP-Example failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is not valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Request ID: 3c714a80-85d1-4166-9c23-a61ed3749abd; Proxy: null)" (RequestToken: 186ba6c7-912e-b3a1-e6f9-408473474a6b, HandlerErrorCode: InvalidRequest)

Reproduction Steps

import os
import sys  # noqa: F401
from aws_cdk import (
    Stack,
    Tags,
    aws_lambda as lambda_,
    aws_stepfunctions as sfn,
    aws_stepfunctions_tasks as tasks,
    aws_events as events,
    aws_secretsmanager as sm,
    SecretValue
)
import aws_cdk as cdk
from constructs import Construct

connection = events.Connection(self, "Connection",
            authorization=events.Authorization.basic("username", SecretValue.unsafe_plain_text("password")))

        task = tasks.HttpInvoke(self, "Invoke HTTP API",
            api_root="States.Format('<endpoint>",  # Static API root, adjust as necessary
            api_endpoint=sfn.TaskInput.from_text("rest/v1/{}.json',$.detail.object_name"), # Use the constructed URL
            body=sfn.TaskInput.from_json_path_at('$.detail.object'),
            connection=connection,
            headers=sfn.TaskInput.from_object({"Content-Type": "application/json"}),
            method=sfn.TaskInput.from_json_path_at('$.detail.object_method'),
            url_encoding_format=tasks.URLEncodingFormat.BRACKETS
        )

        sfMarketo_sync = sfn.StateMachine(
            self,
            "sfMarketo-sync",
            state_machine_name="sfMarketo-sync-test",
            definition_body=sfn.DefinitionBody.from_chainable(task),
            timeout=cdk.Duration.minutes(5),
            state_machine_type=sfn.StateMachineType.STANDARD,
            #role=role
        )
cdk synth
cdk deploy

Possible Solution

When we run synth or deploy on backend it creates cloud formation template in cdk.out folder. The template should add ".$" suffix to "ApiEndpoint" parameter but it's not adding. For reference see the parameter "Method" which has suffix ".$". if it add that suffix it should work.

"EXAMPLERESOURCESsfMarketosync1464C05F": {
   "Type": "AWS::StepFunctions::StateMachine",
   "Properties": {
    "DefinitionString": {
     "Fn::Join": [
      "",
      [
       "{\"StartAt\":\"Invoke HTTP API\",\"States\":{\"Invoke HTTP API\":{\"End\":true,\"Type\":\"Task\",\"Resource\":\"arn:",
       {
        "Ref": "AWS::Partition"
       },
       ":states:::http:invoke\",\"Parameters\":{\"**ApiEndpoint**\":\"States.Format('<endpoint>/rest/v1/{}.json',$.detail.object_name\",\"Authentication\":{\"ConnectionArn\":\"",
       {
        "Fn::GetAtt": [
         "EXAMPLERESOURCESConnectionF6D008B6",
         "Arn"
        ]
       },
       "\"},\"**Method.$**\":\"$.detail.object_method\",\"Headers\":{\"Content-Type\":\"application/x-www-form-urlencoded\"},\"RequestBody.$\":\"$.detail.object\",\"Transform\":{\"RequestBodyEncoding\":\"URL_ENCODED\",\"RequestEncodingOptions\":{\"ArrayFormat\":\"BRACKETS\"}}}}},\"TimeoutSeconds\":300}"
      ]
     ]
    },

Additional Information/Context

No response

CDK CLI Version

2.146.0 (build b368c78)

Framework Version

No response

Node.js Version

v22.3.0

OS

Mac OS

Language

Python

Language Version

Python 3.12.3

Other information

No response

ashishdhingra commented 1 week ago

Reproducible.

Below is the equivalent code in TypeScript:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as events from 'aws-cdk-lib/aws-events';
import * as tasks from 'aws-cdk-lib/aws-stepfunctions-tasks';
import * as sfn from 'aws-cdk-lib/aws-stepfunctions';
export class TypescriptStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const connection = new events.Connection(this, 'Connection', {
      authorization: events.Authorization.basic('username', cdk.SecretValue.unsafePlainText('password'))
    });

    const task = new tasks.HttpInvoke(this, "Invoke HTTP API", {
      apiRoot:"States.Format('https://api.stripe.com')",  // Static API root, adjust as necessary,
      apiEndpoint: sfn.TaskInput.fromText("rest/v1/{}.json',$.detail.object_name"), // Use the constructed URL
      body: sfn.TaskInput.fromJsonPathAt('$.detail.object'),
      connection: connection,
      headers: sfn.TaskInput.fromObject({"Content-Type": "application/json"}),
      method: sfn.TaskInput.fromJsonPathAt('$.detail.object_method'),
      urlEncodingFormat: tasks.URLEncodingFormat.BRACKETS
    });

    const sfMarketo_sync = new sfn.StateMachine(this, 'sfMarketo-sync', {
      stateMachineName: 'sfMarketo-sync-test',
      definitionBody: sfn.DefinitionBody.fromChainable(task),
      timeout: cdk.Duration.minutes(5),
      stateMachineType: sfn.StateMachineType.STANDARD
    });
  }
}

Running cdk synth produces the following output:

Resources:
  Connection07624BCD:
    Type: AWS::Events::Connection
    Properties:
      AuthParameters:
        BasicAuthParameters:
          Password: password
          Username: username
      AuthorizationType: BASIC
    Metadata:
      aws:cdk:path: TypescriptStack/Connection/Connection
  sfMarketosyncRole95111CA2:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: states.amazonaws.com
        Version: "2012-10-17"
    Metadata:
      aws:cdk:path: TypescriptStack/sfMarketo-sync/Role/Resource
  sfMarketosyncRoleDefaultPolicy68A30164:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Statement:
          - Action: events:RetrieveConnectionCredentials
            Effect: Allow
            Resource:
              Fn::GetAtt:
                - Connection07624BCD
                - Arn
          - Action:
              - secretsmanager:DescribeSecret
              - secretsmanager:GetSecretValue
            Effect: Allow
            Resource:
              Fn::GetAtt:
                - Connection07624BCD
                - SecretArn
          - Action: states:InvokeHTTPEndpoint
            Condition:
              StringLike:
                states:HTTPEndpoint: States.Format('https://api.stripe.com')*
            Effect: Allow
            Resource: "*"
        Version: "2012-10-17"
      PolicyName: sfMarketosyncRoleDefaultPolicy68A30164
      Roles:
        - Ref: sfMarketosyncRole95111CA2
    Metadata:
      aws:cdk:path: TypescriptStack/sfMarketo-sync/Role/DefaultPolicy/Resource
  sfMarketosync8373672D:
    Type: AWS::StepFunctions::StateMachine
    Properties:
      DefinitionString:
        Fn::Join:
          - ""
          - - '{"StartAt":"Invoke HTTP API","States":{"Invoke HTTP API":{"End":true,"Type":"Task","Resource":"arn:'
            - Ref: AWS::Partition
            - :states:::http:invoke","Parameters":{"ApiEndpoint":"States.Format('https://api.stripe.com')/rest/v1/{}.json',$.detail.object_name","Authentication":{"ConnectionArn":"
            - Fn::GetAtt:
                - Connection07624BCD
                - Arn
            - '"},"Method.$":"$.detail.object_method","Headers":{"Content-Type":"application/x-www-form-urlencoded"},"RequestBody.$":"$.detail.object","Transform":{"RequestBodyEncoding":"URL_ENCODED","RequestEncodingOptions":{"ArrayFormat":"BRACKETS"}}}}},"TimeoutSeconds":300}'
      RoleArn:
        Fn::GetAtt:
          - sfMarketosyncRole95111CA2
          - Arn
      StateMachineName: sfMarketo-sync-test
      StateMachineType: STANDARD
    DependsOn:
      - sfMarketosyncRoleDefaultPolicy68A30164
      - sfMarketosyncRole95111CA2
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Metadata:
      aws:cdk:path: TypescriptStack/sfMarketo-sync/Resource
...

The "ApiEndpoint":"States.Format('https://api.stripe.com')/rest/v1/{}.json',$.detail.object_name" doesn't appear to be correct since per HTTP Task fields, ApiEndpoint using intrinsic functions should be of format "ApiEndpoint.$":"States.Format('https://api.stripe.com/v1/customers/{}', $.customer_id)".

This could be related to the other issue https://github.com/aws/aws-cdk/issues/29925 as well.