error code when test failing

[afdevries]

Hi, thanks for writing this! I like the clean yaml approach to writing tests. So I started to do a little testing to see how this would fit in our processes for API validation / testing.

When running a simple test when I get an HTTP 400+ response I would expect this would result in setting the exit code in terminal.

Success

version: 1

requests:
  get-test-todo:
    url: https://jsonplaceholder.typicode.com/todos/1
    method: GET

_ strest test.yml

[ Strest ] Found 1 test file(s)
[ Strest ] Schema validation: 1 of 1 file(s) passed

_ Testing get-app-version succeeded

[ Strest ] _  Done in 0.131s

_
_ echo $?
0

Failure

version: 1

requests:
  get-test-todo:
    url: https://jsonplaceholder.typicode.com/todos/230
    method: GET

_ strest test.yml

[ Strest ] Found 1 test file(s)
[ Strest ] Schema validation: 1 of 1 file(s) passed

_ Testing get-test-todo succeeded
_ Testing get-more-todo failed

Error: Request failed with status code 404

[ Strest ] _  Done in 0.453s

_
_ echo $?
0

If there are other good ways to act on failing a test I am open for suggestions.

Thanks again for this project.

[eykrehbein]

Hi and thanks for your issue! You're right, the failure of a request doesn't lead to the end of the process but after a request failed the test function will return and no other request will be sent. So in the end, I think this will end up with the same result. However, I'm open to a discussion, so if you see benefits in exiting the process instantly please drop your thoughts!

[afdevries]

@eykhagen it was not really about it having to fail directly, which might be an option one could set when calling a test run. My post was about the way it handles exit status.

https://shapeshed.com/unix-exit-codes/

The way I would use this is in a pipeline that through the exit code would let me know if things executed succesfully.

[eykrehbein]

Integrating TravisCI

[eykrehbein]

Exit codes added in #8! If a test fails, the process will exit with exit code 1. If everything is fine, it'll exit with code 0.