search

dollabs / pamela

Probabalistic Advanced Modeling and Execution Learning Architecture
Apache License 2.0
233 stars 13 forks source link

Added pseudo fixture for handing stdout/stderr for tests #96

Closed tmarble closed 7 years ago

tmarble commented 7 years ago

A problem with adding "negative" tests is that we expect errors and, previously, these actions would blurt error messages (typically on stderr).

This fix introduces the #'match-eval-out-err macro which can be used it tests to:

  1. Verify the expected exit code (or arbitrary return value)
  2. (Optionally) ensure stdout contains a regex.
  3. (Optionally) ensure stderr contains a regex.

In addition to testing for a failure exit code this macro helps ensure that for both positive and negative test cases that the output has (at least) some sanity as represented by the given regex and neither stream pollutes the test output (which could lead to confusion about the overall success of tests).

Signed-off-by: Tom Marble tmarble@info9.net

tmarble commented 7 years ago

In addition the main cli.clj change changes the output

17-04-27 19:34:50 cerise WARN [pamela.cli:288] - exit 0 pamela in DEV MODE. Not exiting repl? false test-mode true

to log/info level to reduce the output pollution when running tests.

pmdoll commented 7 years ago

Should we still expect to see WARN instead of INFO for these messages? WARN exit 0 plan-schema in DEV MODE. Not exiting -> repl? false test-mode true

tmarble commented 7 years ago

@pmdoll no, it adds stdout pollution... see the comment 2 above.