42ShellTester

42ShellTester is an integration testing framework wrote in Bash and designed for the pedagogical projects of the Shell branch at School 42 (Paris) listed bellow:

minishell
21sh
42sh

It brings you an easy way to add, maintain and run integration tests, helping you to work step by step on your Shell implementation.

42ShellTester is currently packaged with 289 tests.

Install

git clone https://github.com/we-sh/42ShellTester ~/42ShellTester

Run tests

Add the path to your Shell as argument:

bash ~/42ShellTester/42ShellTester.sh "/ABSOLUTE/PATH/TO/YOUR/SHELL"

Options

`--filter` + `$regex`

Run tests that matches with the specified regular expression (e.g. --filter "builtins").

`--reference` + `$binary`

Run tests that does not fail with the specified Shell binary (e.g. --reference "bash").

`--posix`

Run tests that are POSIX compliant only (run all by default).

`--hard`

Run tests that are marked as « hard » (omitted by default).

`--pending`

Also run pending tests.

`--all`

Equivalent to use the two options --pending and --hard together.

`--show-success`

Also display tests that succeed (hidden by default).

List of tests

Development

Coding convention

Scope indentation must be done with 2 spaces
Variable names must be upper case (e.g. INDEX)
Global variables must be prefixed with GLOBAL_ (e.g. GLOBAL_TOKEN)
Variable expansion must be surrounded by curly braces (e.g. ${VARIABLE})
Arguments of functions and commands must be surrounded by double or simple quotes (e.g. run_assert "STDERR")
Semicolon are banned so that words like then and do are always at start of lines (e.g. wrong inline code: if [ ... ]; then cmd; fi)
Folder names may only contain alphanumeric and -

Adding new test

An integration test must be self-sufficient, that means executing the full test suite or only one test must result in the same failed or success status. The framework 42ShellTester brings you tools for that!

Firstly, tests are executed inside a temporary folder tmp/ that is created at launch time and placed at the root installation folder of the framework. You may generate temporary files, binaries and folders that are needed for your test, but pay attention to not touch external folders. Use the before_exec callback to generate these resources.

Secondly, each test is executed within a sub-shell, so that you may modify the environment without disrupting the test suite. Use the before_exec callback to modify the environment.

Thirdly, a test must concern one single feature at a time, that means wherever possible you must avoid the use of multiple builtins or capabilities (e.g. do not use a pipe | within a test that concerns the builtin env, or again use absolute paths to binaries like /bin/ls to let the Shell implementation not support the PATH, except if you precisely test this feature!).

Fourthly, when a test need binaries like /bin/env or /bin/echo, prefer to recode your own, simplier and multi-platform, and place it in support/ folder. Then use the before_exec callback to compile it and make it available for your test.

Sixthly, a test that is not POSIX compliant must contain a file named non-posix containing a small explanation of why.

Finally, don't write a README and let the task generate_readmes do it for you :-) A description may be added in a file named description that will appear at the top of the README.

Follow the guideline to add a new test:

Create a sub-folder in spec/ (e.g. spec/minishell/builtin/cd/new-test/)
If necessary, create a file before_exec that contains the shell commands that prepare the environment and the temporary resources (e.g. mkdir valid_folder)
Create a file stdin that contains the shell command you want to test (e.g. cd invalid_folder)
Create the files stdout and/or stderr that contain the expected output assertions (e.g. in stderr: expected_to_not be_empty) (see available assertions and verbs bellow)
You may also create a file misc that contains special expectations not concerning output on standard and error (e.g. expected_to_not exit_with_status 0)
If necessary, create a file description that describes more precisely the purpose of the test (e.g. Trying to access invalid folder must display an error on standard error and result in a failure status code) (the description will be included at top of the auto-generated README)
If the test is not POSIX compliant, create a file non-posix that explains why.

Assertions

expected_to / expected_to_not + verb: An assertion beginning with expected_to (or its opposite expected_to_not) makes the test resulting in failure status if the expectation that follows does not comply.
might / might_not + verb: An assertion beginning with might (or its opposite might_not) always makes the test resulting in success status. When the expectation that follows may not comply, it is nevertheless considered as success but it displays a warning message.

Verbs

be_empty: Actual output is empty.
create_file + $filename: Actual command creates a file named $filename. May also be followed with a file test:
- matching_regex + $regex: At least one line of the file matches with the regular expression $regex.
- not_matching_regex + $regex: Any line of the file does match with the regular expression $regex.
- with_nb_of_lines + $int: The file contains exactly $int lines.
exit_with_status + $int: The Shell termination results in the exit status $int.
have_nb_of_lines + $int: Actual output contains exactly $int lines.
match_regex + $regex: At least one line of actual output does match with the regular expression $regex.
- once: The matching is limited to only one occurrence.
- $int times: The matching must exactly occur $int times.
match_each_regex_of_file + $filename: Actual output does match with each regular expression contained in the file named $filename (in an indifferent order).

Adding new verb

A verb is a function that is prefixed by run_verb_ and that returns 0 or 1 according to the tested behavior. It may return a status 255 when bad or missing argument.

At runtime, the framework provides a list of variables that can be used by the verbs:

RESPONSE: The path to the file containing actual output (STDOUT or STDERR)
RESPONSE_EXIT_STATUS: The exit status of the Shell termination
EXPECTED_TO_ARGS[]: An array containing the arguments following the verb

Follow the guideline to add a new verb:

Choose the best name that respects the CamelCase convention and that can be human-readable when used with an assertion (e.g. expected_to be_empty can be read actual output is expected to be empty)
Create a file in lib/verbs/ with the exact name of the verb and that is prefixed with run_verb_ (e.g. lib/verbs/run_verb_be_empty.sh
Add a shebang: #!/bin/sh and a comment that describes the tested behavior
Create a function with the exact name of the verb and that is prefixed with run_verb_ (the same as the file name) and make it respect the following rules:
- Local variables must be declared with local
- No output can be done with echo or printf
- Function returns 0 on succes, 1 on fail or 255 on bad use
- Use the array EXPECTED_TO_ARGS[] to take advantage of arguments (e.g. expected_to match_regex "regex", then EXPECTED_TO_ARGS[0] contains regex)

Support binaries

The framework 42ShellTester provides several binaries to be used within the tests. Using them instead of using Unix binaries prevents from undefined behaviors and compatibility errors.

Find the available list of support binaries bellow:

./display_env: A binary that iterates on **envp and write each element on standard output.
./display_program_name: A binary that writes its name on standard ouput.
./display_pwd: A binary that writes on standard output the absolute path of the current directory returned by getcwd(3), encountered with the strings PWD: and :PWD.
./exit_with_status: A binary that immediately exits with the status given as first argument.
./read_on_stdin: A binary that reads on standard entry and write each line on standard output suffixed with the character @ (e.g. same behavior as cat -e and the newline character). When read(2) returns -1, then the string STDIN READ ERROR is written on standard error.
./sleep_and_exit_with_status: A binary that sleeps for a duration in seconds given as first argument and then exits with status given as second argument.
./sleep_and_write_on_stderr: A binary that sleeps for a duration in seconds given as first argument and then writes on STDERR the string given as second argument without EOL.
./write_all_arguments_on_stdout: A binary that writes on standard output each argument separated by the symbol @. If no argument is given, it writes the string "nothing to be written on stdout".
./write_on_stderr: A binary that writes on standard error the first given argument (the same behavior as echo but with only one argument) and exits with an error status code given as second argument. If no argument is given, it writes the string "write on stderr" and exit with status 1.
./write_on_stdout: A binary that writes on standard output the first given argument (the same behavior as echo but with only one argument). If no argument is given, it writes the string "write on stdout".
./write_on_stdout_and_stderr: A binary that writes on standard output the first given argument, and writes on standard error the second given argument. If an argument is missing, it writes the strings "write on stdout" and "write on stderr".

Tasks

bash ./tasks/generate_readmes.sh (only on master branch) to automaticaly generate the README files of tests

The Team

Adrien Nouvel @anouvel
Gabriel Kuma @gabkk
Jean-Michel Gigault @jgigault

Logo credits

Edouard Audeguy
Illustrateur / Infographiste
https://edouardaudeguy.wix.com/portfolio

we-sh / 42ShellTester

readme