search

dlang / dub

Package and build management system for D
MIT License
670 stars 230 forks source link

No documentation for non-unittest tests #1697

Open Samathy opened 5 years ago

Samathy commented 5 years ago

How to go about writing tests which live in the test/ directory isnt documented, leaving contributors to learn from the existing examples. This could be improved with a short note to perhaps CONTRIBUTING.md, an addition to the dlang-bot automatic PR comment or a README.md in the test directory.

Below is a suggestion as to the initial content of this note, however as a new contributor I cannot in any way expect this to be correct or complete. Its just what I've learned over the past couple of days from reading the examples and trial and error with the CI system(s).

Full-Stack tests may be written for any given bug, they live in the test directory`.

A test must be named using the issueXXXX-thing-fails-for-thing convention. A bash script ( with executable flag set ) and folder should be created with this name.

The script may execute arbitrary commands designed to replicate the bug in question, it may use the folder of the same name in any way to support recreating this bug. Scripts have a number of special variables they may access:

$DC    The D compiler
$DUB    The built dub executable
$CURR_DIR    The current directory
$HOME    The home directory of the user

Scripts should use $DUB to run the dub executable built from the current branch being tested. Scripts should use $DC to set the compiler Dub uses. Don't worry about explicitly setting a compiler ( i.e dmd, ldc, or gdc ), each test script is ran with each compiler. The $(HOME) path should be used to avoid stomping over files which should not be stomped over.

Test scripts are free to create and delete files almost everywhere ( i.e userwide dub configurations ) in order to replicate their bug.

A test's corresponding directory is commonly used to contain an example dub project.

Tests in the test directory should be used to demonstrate bugs which are exposed by things like:

Try reading the existing tests in the test directory. Please include a test if you can.

Samathy commented 5 years ago

Additional content

Tests can be ran locally by exporting those special variables for your local environment. For example, if the script you need to run uses CURR_DIR and DUB. Set those environment variables to point to something reasonable before running the script.

e.g export CURR_DIR=. export DUB=../bin/dub

Be careful though, some test scripts modify systemwide or userwide settings! Read the script before running it, or run it in a sandboxed environment.