search

bluebird75 / luaunit

LuaUnit is a popular unit-testing framework for Lua, with an interface typical of xUnit libraries (Python unittest, Junit, NUnit, ...). It supports several output formats (Text, TAP, JUnit, ...) to be used directly or work with Continuous Integration platforms (Jenkins, Maven, ...).
Other
565 stars 136 forks source link

Allow the output format to be specified by the environment #133

Closed kevans91 closed 3 years ago

kevans91 commented 3 years ago

Described in a little more in-depth in the commit messages, but this small PR would be quite appreciated -- this will check for LUAUNIT_DEFAULT_OUTPUT in the environment and use that as if it were supplied as the --output option in the absence of an overriding --output option from the arguments. This opens us up to using a shebang line to fully specify how the test is generally desired to be read:

#!/usr/bin/env -S LUAUNIT_DEFAULT_OUTPUT=TAP /usr/local/bin/lua53

Specifically, I'd like to import luaunit into FreeBSD's base system for unit testing of the lua bindings we're writing for various base libs, and our infrastructure can/will insert effectively the above shebang (except using our base lua interpreter) so that the test driver (Kyua) can just execute this test like it does any other test and examine the output without having to understand that it's a lua script or anything else.

A functional smoke test has been added to insert LUAUNIT_DEFAULT_OUTPUT into the environment for one of the TAP tests that's already passed to try and make it clear if any failure is due to a regression in the TAP output vs. the new environment variable. TAP was randomly chosen for this example because it was the first non-default format test that I came across.

kevans91 commented 3 years ago

It looks like I'll need to reimagine the test change, since I forgot the test needs to be able to run on Windows as well...

bluebird75 commented 3 years ago

Nice. Can you tell me a little bit more about what you are testing with luaunit, out of curiosity ?

kevans91 commented 3 years ago

Nice. Can you tell me a little bit more about what you are testing with luaunit, out of curiosity ?

Sure! We imported Lua into the base distribution of FreeBSD, and now we're writing bindings for various system libs so we can write some utilities in Lua. e.g. we produced some bindings for libjail that does operations on jails (much like containers, if you're not familiar), and we're needing tests for the various bindings we're producing before we start writing system utilities that depend on them.

bluebird75 commented 3 years ago

I am interested in more feedback on using lua with many tests. I believe the current setup of declaring global tests is osolete and also luaunit should provide more convenience on running a test module with one or several test cases.

Don't hesitate to share your ideas or paint point in the process !

kevans91 commented 3 years ago

Sure, will do! I tend to agree- our C/C++/sh testing frameworks all typically require you to specify the test to run and also drive cleanup independently -- I tend to not care about driving cleanup independently, but it is nice to be able to run a single test case in isolation.

The downside is their output formats are really bad. I really like luaunit's default output format and how it actually provides useful information about failures.

bluebird75 commented 3 years ago

Note that I changed the variable to LUAUNIT_OUTPUT for more simplicity.