When validate=true, Map state machines in DISTRIBUTED mode incorrectly throw a yaml validation error

[robtoth]

This is a (Bug Report / Feature Proposal)

Description

Attempting to deploy a state machine with Map in Distributed mode: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-asl-use-map-state-distributed.html

serverless.yml

...(events, etc.)
      definition:
        StartAt: MyMapState
        States:
          MyMapState:
            Type: Map
            ItemProcessor:
              StartAt: My map task
              States:
                My map task:
                  Type: Task
                  Resource:
                    Fn::GetAtt: [mylambda, Arn]
                  End: true
              ProcessorConfig:
                Mode: 'DISTRIBUTED'
                ExecutionType: 'STANDARD'
            End: true
  validate: true

This fails:

**$ sls deploy**
Running "serverless" from node_modules

Deploying project to stage dev (us-east-1)

✖ Stack project failed to deploy (18s)
Environment: linux, node 12.22.9, framework 3.24.1 (local) 3.24.1v (global), plugin 6.2.2, SDK 4.3.2
Credentials: Local, "semler-iac" profile
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
✕ State machine "MyMapState" definition is invalid:
SCHEMA_VALIDATION_FAILED: /States/MyMapState is invalid. must match exactly one schema in oneOf

However, when I turn the validator off and let it try to update on AWS cloudformation, it succeeds:

...(events, etc.)
      definition:
        StartAt: MyMapState
        States:
          MyMapState:
            Type: Map
            ItemProcessor:
              StartAt: My map task
              States:
                My map task:
                  Type: Task
                  Resource:
                    Fn::GetAtt: [mylambda, Arn]
                  End: true
              ProcessorConfig:
                Mode: 'DISTRIBUTED'
                ExecutionType: 'STANDARD'
            End: true
  validate: false

$ sls deploy
Running "serverless" from node_modules

Deploying semler-measurement-quality to stage dev (us-east-1)

✔ Service deployed to stack semler-measurement-quality-dev (121s)

Notes

It's important to realize that when in distributed map state, "Iterator" must be replaced with "ItemProcessor" https://docs.aws.amazon.com/step-functions/latest/dg/concepts-asl-use-map-state-distributed.html#map-state-distributed-additional-fields

I'm assuming the validator here in this plugin https://github.com/serverless-operations/serverless-step-functions/blob/42bd42350c33c418762bff6f75cb8b2d6e2d3729/lib/deploy/stepFunctions/compileStateMachines.js#L120 just pulls calls the main serverless validator function here https://github.com/serverless/serverless/blob/906ea319dd1486f79d6e088a999fd5634526c4bc/lib/classes/config-schema-handler/index.js#L77 which somehow compares the known schema to something (pulled from AWS?).

That's as far as I was able to dig into the code. I couldn't find where the schema for the "ItemProcessor" yaml section was being defined, perhaps on AWS cloud or perhaps I just couldn't find it in the code.

But regardless, it seems that the validator expects only Iterator instead of ItemProcessor and therefore will not allow the user to use the map in distributed mode.

Additional Data

$ sls --version
Running "serverless" from node_modules
Framework Core: 3.24.1 (local) 3.24.1 (global)
Plugin: 6.2.2
SDK: 4.3.2

cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"

cat package.json 
{
  "devDependencies": {
    "serverless": "^3.22.0",
    "serverless-iam-roles-per-function": "^3.2.0",
    "serverless-python-requirements": "^6.0.0",
    "serverless-step-functions": "^3.11.1"
  }
}

(Error messages provided above)

[robtoth]

It's also of note that the step function workflow studio seems to incorrectly puts Iterator instead of ItemProcessor ?

[robtoth]

Here is an example using ItemProcessor: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-asl-use-map-state-distributed.html#distributed-map-example

[robtoth]

When turning off the validator and letting it deploy to cloudformation, it correctly defines the yaml for distributed state.

This is also carried over to the workflow studio editor

So the issue seems to just be the validation using the incorrect schema when trying to use map in distributed mode.

[Yitzchak-SB]

Hi, i encountered the same issue.

When deploying with validate=false or through AWS Console everything is deployed. Setting validate=true fails the deployment with SCHEMA_VALIDATION_FAILED: /States/MyMapState is invalid. must match exactly one schema in oneOf

[robtoth]

Hi, i encountered the same issue.

When deploying with validate=false or through AWS Console everything is deployed. Setting validate=true fails the deployment with SCHEMA_VALIDATION_FAILED: /States/MyMapState is invalid. must match exactly one schema in oneOf

Glad to hear it's replicatable! Do you have any idea where the schema is being pulled from to validate it, and why it's failing @Yitzchak-SB ?

[akshaydk]

I'm facing the same issue. It replicable because the current release do not support the latest changes from Step Functions. I turned off the validation for now. Here's the info of the currently supported attributes from AWS.

[rajnish307]

@Yitzchak-SB @akshaydk Is validate option added in latest version? I am using these configurations and its throwing error as validate keyword is not allowed.

Error: ServerlessError: State machine [workflow] is malformed. Please check the README for more info. ValidationError: "validate" is not allowed

My Configuration: Operating System: linux Node Version: 14.21.3 Framework Version: 2.55.0 (local) Plugin Version: 5.4.3 SDK Version: 4.3.0 Components Version: 3.15.1

Edit: This issue was fixed after updating my serverless version

[scottgit]

As of 3.13.1 it is still failing validations. I am still getting an issue trying to include ItemProcessor:

State machine "sentry-stepfunction-test" definition is invalid:
SCHEMA_VALIDATION_FAILED: /States/makeMap/ItemProcessor is invalid. must match exactly one schema in oneOf

[luksfarris]

I believe this is working on 3.18.0. At least I got it to work. Initially I got the error:

/States/[State Name]/ItemProcessor/ProcessorConfig/Mode is invalid. must be equal to one of the allowed values

But the problem was in me setting ProcessorConfig.ExecutionType to Standard, when the correct value is STANDARD. Setting validate: false under stepFunctions: helped me debug the issue, because the CloudFormation error was more clear.

Looking at what was actually created in AWS, it looks correct

"SomeMap": {
      "Type": "Map",
      "ItemsPath": "$.Response",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "DISTRIBUTED",
          "ExecutionType": "STANDARD"
        },
        "StartAt": "SomeTask",
        "States": {
          "SomeTask": {
            "Type": "Pass",
            "Result": "$$.Execution.Input",
            "End": true
          }
        }
      },
      "End": true
    }
  }