search

awslabs / states-language

The States Language spec
https://states-language.net/spec.html
22 stars 17 forks source link

Publish test vectors for ASL #27

Open ZackKanter opened 2 years ago

ZackKanter commented 2 years ago

Specifications often include test vectors to ensure consistent implementation across languages. Those test vectors are then referenced in libraries that implement the spec. If these were available, it would be extremely helpful for developers working with Amazon States Language.

timbray commented 2 years ago

Um, nice stuff, but it's not instantly obvious to me how you'd use these in working on Amazon States Language.

ZackKanter commented 2 years ago

Um, nice stuff, but it's not instantly obvious to me how you'd use these in working on Amazon States Language.

One way to use these would be while writing something like ts2asl – if all of the test cases passed, we would know that we supported the entire spec.

OlafConijn commented 2 years ago

I think open-sourcing test vectors would be a great addition to the documentation at states-language.net.

This would have helped quite a bit in understanding some of the behaviors that is not documented, two examples come to mind:

JsonPath, the language spec mentions support for JsonPath, however, it links to another github repo of which part is supported. There is no clear way of telling what feature is (or isn't) supported.

Intrinsic Functions, the language spec doesn't provide a lot of detail. Example, for States.StringToJson it reads "The interpreter applies a JSON parser to the Value and returns its parsed JSON form." but fails to document the behavior of e.g. States.StringToJson("20") which returns a number.

Documenting all of JsonPath or Intrinsic functions might not have a place in the language spec for States Languages. accompanying the spec with tests (or additional documentation?) would however be useful. I would prefer to have the tests over documentation as it would be more concise. It would also be a great way to discuss improvements (create a PR with a test that could be used as a bug report or feature request).