comps / ptef

Portable Test Execution Framework
MIT License
5 stars 0 forks source link

Add support for non-standard exit codes #2

Closed comps closed 2 years ago

comps commented 2 years ago

The practice of test suites using exit codes of tests to mean SKIP / ERROR / CONF / WARN / etc. is pretty widespread. Due to the lack of any standardization, the PTEF specification doesn't go beyond 0=PASS, else=FAIL, ... however the reference implementation should probably allow some customization, like with result status colors.

Unlike status colors, exit codes should likely be a simple static 256-member-long array of char*, defining statuses for individual exit codes, with 0/NULL meaning "use default". Then there should be another global char* variable defining the default status.

char *ptef_status_codes[256] = {
    [0] = "PASS",
    [2] = "WARN",
    [10] = "ERROR",
};
char *ptef_status_codes_default = "FAIL";

The standard-specified default would then be

char *ptef_status_codes[256] = {
    [0] = "PASS",
};
char *ptef_status_codes_default = "FAIL";
comps commented 2 years ago

https://github.com/comps/ptef/commit/2950d2b065904085ff394d515845c27d1acee043