Add a more formal test suite to es

pmarreck commented 2 years ago

Currently, es has been "tested" by running trip.es, but this is very insufficient. Contributors would be averse to making any serious feature-additions unless there was a good test suite to verify they didn't break anything important. I propose simply initializing such a test suite, written in es, that does something simple such as run one of the examples and assert on the standard-out and return code of the result (we can always add to it later once the structure is in place!)

In order to accomplish that, we also (ideally) need some sort of assert library written in es which captures those things from running any es command and allows us to do things like assert_equal, assert_not_equal, assert_success, assert_failure, assert_match and assert_no_match on any of stdout, stderr or return codes. Alternately, we could use an assert/test library written in another scripting language such as bash and run es code and assert on it from there. (I looked at a few- BATS is basically dead, and ShellSpec is full-featured but uses cucumber-esque syntax and since cucumber is gross (is it just me, or are API's which attempt to look like English, but are not English, basically the "uncanny valley" of API's?), I am EXTREMELY averse to using it). But it might make sense to bootstrap a suite with one of those (or another) with an eye towards replacing it with pure es in the future. Thoughts?

jpco commented 1 year ago

I wrote up a basic es-based test harness a while ago. See the https://github.com/jpco/es-shell-mirror/tree/master/test directory.

Awkwardly, I mostly made this version in order to support testing my arithmetic syntax which (see other discussion in this repo) nobody else wants, so I would have to do some reworking to remove that. I'd also need to vastly improve the documentation -- the behavior of the test variable at the top of each test file is really opaque, and is pretty core to how the whole harness functions.

I think what I wrote is a useful starting place, though -- especially since I already went through the annoyance of hardening it against segfaults and other catastrophes. What do folks think?

The idea is that there's a central file test.es that goes through the tests directory and runs each test file. The test files are es scripts which

set a variable test to a lambda which controls how to run this particular set of test cases, and then
run case and want to create the individual cases and desired results.

There's also a skip function which is helpful for recording cases that you'd like to work later but don't expect to work now.

An example is probably most helpful here. I wrote the following test to make sure = would be parsed how I wanted. It uses a pre-defined get-output test lambda, but other tests have custom-written code in test to extract success/failure, output on a fd, or a return value. I imagine a set of pre-defined test functions could fit well with your ideas for assert_* functions.

#! /usr/local/bin/es --

# Tests that these cases parse '=' correctly, either as assignment
# or as a literal (equivalent to a quoted '=')
test = get-output

case 'a=b; echo $a'                   ; want 'b'
case 'echo a=b'                       ; want 'a=b'
case 'echo a=b; a=b; echo $a'         ; want 'a=b b'
case 'a =b= c==d= e=f; echo $a'       ; want 'b= c==d= e=f'
case '''a=b'' = c=d; echo $''a=b'''   ; want 'c=d'
case 'a=b = c = d; echo $a'           ; want 'b = c = d'
skip case 'echo a = b'                ; # want 'a = b'

pmarreck commented 1 year ago

That's pretty smart, and certainly way better than the current situation!

After I wrote that comment, I ended up writing https://github.com/pmarreck/tinytestlib in Bash (which can of course test anything commandline-driven, but in this case wouldn't be eating our own dogfood...) I felt it was key to somehow capture {stdout, stderr, returncode} as a tuple-ish thing (well, an associative array in this case), from a single run of a command, since in some cases you'd want to assert on more than 1 of those in a suite. I don't know if this is possible to do in es-shell; the Bash solution involved this minor monstrosity https://github.com/pmarreck/tinytestlib/blob/yolo/tinytestlib#L141 which in essence reroutes those values in turn through stdout so they can be assigned to interim variables and then collected.

wryun / es-shell

Add a more formal test suite to es #35