Comcast / plax

A test automation engine for messaging systems
Apache License 2.0
7 stars 14 forks source link

Feature Request - Plax CLI should error out if you send a Plaxt test request without supplying all the parameters required in a test #70

Open t-harwood opened 3 years ago

t-harwood commented 3 years ago

Plax should return an error if a plax test contains variables that are not defined / supplied in the plax test run command.

For example, a test may contain a section like this:

        payload:
          url: '{?!HOSTANDPORT}/api/{?!ACCOUNTID}?userid={?!USERID}'
          method: GET

If you then send the plax test run command without supplying those variables, the test currently runs and fails without an error explaining that those variables weren't defined.

Bad request that should produce an error: plax -test get-account.yaml -p '?!HOSTANDPORT=http://localhost:9090' -log debug

Properly formatted request: plax -test get-account.yaml -p '?!HOSTANDPORT=http://localhost:9090' -p '?!ACCOUNTID'='12345abcde' -p '?!USERID'='test-users' -log debug

jsccast commented 3 years ago

The commit referenced above includes some explanatory comments.

When doing string-based substitutions, an unbound var should (at least in conventional settings) result in an error. The pending PR #64 works this way, and the commit above adds an explicit unit test.

Note that in the context of a recv, a var subject to structural substitution ({"send":"?X"}) that isn't bound is not an error. The whole purpose could be to get a binding for that var. Without additional information, Plax would have a hard time knowing when to complain. The whole topic of analyzing/learning what structural variables should be bound when is interesting and challenging.

jsccast commented 3 years ago

When doing string-based substitutions, an unbound var should (at least in conventional settings) result in an error.

Maybe not -- unless recv regular expressions are unconventional.

We ran into

recv:
  regexp: param {FOO}={?P<BAR>.*}

that was attempting to parse lines like param {FOO}=foo and then bind ?BAR. We want {FOO} and {?P<BAR>.*} to be interpreted literally.

Probably should re-open so that an unbound var in a regular expression in a recv does not result in an error.

jsccast commented 3 years ago

Possible work-around, which might even be the right approach: Use blackslashes to escape the braces in the regular expression:

recv:
  regexp: param \{FOO\}={?P<BAR>.*}`

Should consider supporting brace escapes outside of regular expressions.