5 - Adding error handler for invalid YAML

samanthakem commented 4 years ago

[x] Does all of your new or changed code have unit tests?
[x] Is overall unit test statement coverage still above 85%?
[x] Is code style linting passing with npm run lint?
[x] Is the related GitHub issue number included in your branch name? ex branch name: fix/12-some-bug
[x] Is the related GitHub issue number included in your pull request title? example PR title: "12 - Fixes some bug"
[x] Is documentation still up to date after your change?

As right now, the fix is basically adding the parseYamlFile as a first step for every file, the step will verify the current yaml, if it fails I get the exception message and add it as the reason for the test to fail which indicates the line and column from the invalid yaml file, if there is an invalid yaml file, the test will stop right away. Check the outputs below:

====================== OUTPUTS =====================

Single invalid yaml file fails:

[INFO]     Executing run step: parseYamlFile
[ERROR]    FAILED "parseYamlFile": bad indentation of a mapping entry at line 5, column 3:
      - /On branch ZZZ-[0-9]{0,10}\/no-id/
      ^ (1ms)

[SUMMARY]  ============= JUDO TESTS COMPLETE =============
[SUMMARY]  test-examples/wrong-yml.yml
[ERROR]    FAILED: parseYamlFile (0.001s)
[SUMMARY]  Total tests: 1 (0.001s)
[SUMMARY]  Passed:      0
[SUMMARY]  Failed:      1

Single file test succeeds:

[INFO]     Executing run step: helloWorld
[INFO]     Executing: echo "hi!"
[STDOUT]   hi!

[INFO]     exited: 0
[SUCCESS]  PASSED "helloWorld" (10ms)
[INFO]     Executing run step: helloWorld2
[INFO]     Executing: echo "hi!"
[STDOUT]   hi!

[INFO]     exited: 0
[SUCCESS]  PASSED "helloWorld2" (4ms)

[SUMMARY]  ============= JUDO TESTS COMPLETE =============
[SUMMARY]  test-examples/hello-world.yml
[SUCCESS]  PASSED: helloWorld (0.01s)
[SUCCESS]  PASSED: helloWorld2 (0.004s)
[SUMMARY]  Total tests: 2 (0.014s)
[SUMMARY]  Passed:      2
[SUMMARY]  Failed:      0

Directory tests fail with both invalid yaml and invalid step output:

[INFO]     Found 7 tests in test-examples/
[INFO]     Executing run step: helloWorld
[INFO]     Executing: echo "hi!"
[STDOUT]   hi!

[INFO]     exited: 0
[SUCCESS]  PASSED "helloWorld" (10ms)
[INFO]     Executing run step: helloWorld2
[INFO]     Executing: echo "hi!"
[STDOUT]   hi!

[INFO]     exited: 0
[SUCCESS]  PASSED "helloWorld2" (4ms)
[INFO]     Executing run step: goodbyeWorld
[INFO]     Running prerequisite commands in: /Users/smonteiro/projects/intuit/judo
[OUTPUT]   this is a prerequisite command

[INFO]     Prerequisites complete
[INFO]     Executing: echo "goodbye!"
[STDOUT]   goodbye!

[INFO]     exited: 0
[SUCCESS]  PASSED "goodbyeWorld" (9ms)
[INFO]     Executing run step: helloMarsShouldFail
[INFO]     Executing: echo "blorp!"
[STDOUT]   blorp!

[INFO]     exited: 0
[ERROR]    FAILED "helloMarsShouldFail": Expected output to contain: goodbye! (5ms)
[INFO]     Executing run step: helloWorld
[INFO]     Executing: echo "hi!"
[STDOUT]   hi!

[INFO]     exited: 0
[SUCCESS]  PASSED "helloWorld" (4ms)
[INFO]     Executing run step: helloWorld
[INFO]     Executing: echo "hi!"
[STDOUT]   hi!

[INFO]     exited: 0
[SUCCESS]  PASSED "helloWorld" (3ms)
[INFO]     Executing run step: helloWorldAgain
[INFO]     Executing: echo "hi again!"
[STDOUT]   hi again!

[INFO]     exited: 0
[SUCCESS]  PASSED "helloWorldAgain" (3ms)
[INFO]     Executing run step: goodbyeWorldAgain
[INFO]     Executing: echo "bye again!"
[STDOUT]   bye again!

[INFO]     exited: 0
[SUCCESS]  PASSED "goodbyeWorldAgain" (4ms)
[INFO]     Executing run step: timeoutPass
[INFO]     Executing: sleep 1s
[INFO]     exited: 0
[SUCCESS]  PASSED "timeoutPass" (1004ms)
[INFO]     Executing run step: timeoutFail
[INFO]     Executing: sleep 2s
[INFO]     exited: 0
[SUCCESS]  PASSED "timeoutFail" (2005ms)
[INFO]     Executing run step: timeoutOngoing
[INFO]     Executing: bash ./test-examples/timeout-suite/timeout-test.sh
[STDOUT]   waiting 1s

[STDOUT]   first sleep done, waiting 1s

[STDOUT]   second sleep done, waiting 3s

[INFO]     exited: 0
[SUCCESS]  PASSED "timeoutOngoing" (5026ms)
[INFO]     Executing run step: parseYamlFile
[ERROR]    FAILED "parseYamlFile": bad indentation of a mapping entry at line 5, column 3:
      - /On branch ZZZ-[0-9]{0,10}\/no-id/
      ^ (1ms)

[SUMMARY]  ============= JUDO TESTS COMPLETE =============
[SUMMARY]  test-examples/hello-world.yml
[SUCCESS]  PASSED: helloWorld (0.01s)
[SUCCESS]  PASSED: helloWorld2 (0.004s)
[SUMMARY]  test-examples/simple-test-suite/goodbye-world-fail.yml
[SUCCESS]  PASSED: goodbyeWorld (0.009s)
[ERROR]    FAILED: helloMarsShouldFail (0.005s)
[SUMMARY]  test-examples/simple-test-suite/hello-world-regex.yml
[SUCCESS]  PASSED: helloWorld (0.004s)
[SUMMARY]  test-examples/simple-test-suite/hello-world.yml
[SUCCESS]  PASSED: helloWorld (0.003s)
[SUMMARY]  test-examples/simple-test-suite/nested-test-suite/hello-goodbye-world-again.yml
[SUCCESS]  PASSED: helloWorldAgain (0.003s)
[SUCCESS]  PASSED: goodbyeWorldAgain (0.004s)
[SUMMARY]  test-examples/timeout-suite/timeout.yml
[SUCCESS]  PASSED: timeoutPass (1.004s)
[SUCCESS]  PASSED: timeoutFail (2.005s)
[SUCCESS]  PASSED: timeoutOngoing (5.026s)
[SUMMARY]  test-examples/wrong-yml.yml
[ERROR]    FAILED: parseYamlFile (0.001s)
[SUMMARY]  Total tests: 12 (8.078s)
[SUMMARY]  Passed:      10
[SUMMARY]  Failed:      2

ejfrancis commented 4 years ago

Adding a special named step seems like a bit of a hack. Getting the line number and preview here is great, but from the user's perspective I'd want the output to be something unique to when parse errors occur to make it more clear

samanthakem commented 4 years ago

Okay so I just added a logger.error(...) which is specifies the issue with yaml parsing + process.exit(1).

The output would be this for a invalid yaml file:

[ERROR]   YAML PARSER FAILED on file test-examples/wrong.yml: bad indentation of a mapping entry at line 5, column 3:
      - /On branch ZZZ-[0-9]{0,10}\/no-id/
      ^

intuit / judo

5 - Adding error handler for invalid YAML #32