Exit with a special code for tool/browser failure

[zzo]

Expected behavior

If karma tests fail due to browser issue karma single run should exit with an exit code to differentiate between a 'tool failure' vs. a 'test failure'. Would like karma to 'exit(100)' to indicate this type of failure or the error code must be configurable for these situations.

Actual behavior

Karma always exits with a '1' status whenever anythinng goes wrong, test failure or browser failure

Enviroment Details

• Karma version (output of karma --version):
• Relevant part of your karma.config.js file

[zzo]

Here is a list of potential errors and suggested exit codes to align with protractor for consistent error reporting:

Error code / Error category 1 Actual test failure 100 Installation corruption / Node / node_modules issues (exception when bootstrapping the tool) 105 Fail to parse config file 110 Plugin error (could be almost anything in karma-land if a plugin fails) 120 Failure during startup/teardown 130 Requested a browser that’s not available 135 Unable to start browser 140 Unable to connect to browser 150 Test framework exception (internal test harness error) 160 Fail to parse tested files 170 Disconnect from browser 180 Test timeout 199 Anything not listed above

[Moumi]

Where would you suggest to change/add these exit codes? I have seen this myself and it is indeed, as you say, not useful to use one exit code for all failures/errors. I would love to work on this, but I would like to know the correct and most useful place to set the exit codes. If it is possible to help me figure this out, I'll start working on it!

[dignifiedquire]

If we do this I would like to adhere to standard exit codes defined in sysexits.h (at least for those that match)

#define EX_OK       0   /* successful termination */

#define EX__BASE    64  /* base value for error messages */

#define EX_USAGE    64  /* command line usage error */
#define EX_DATAERR  65  /* data format error */
#define EX_NOINPUT  66  /* cannot open input */
#define EX_NOUSER   67  /* addressee unknown */
#define EX_NOHOST   68  /* host name unknown */
#define EX_UNAVAILABLE  69  /* service unavailable */
#define EX_SOFTWARE 70  /* internal software error */
#define EX_OSERR    71  /* system error (e.g., can't fork) */
#define EX_OSFILE   72  /* critical OS file missing */
#define EX_CANTCREAT    73  /* can't create (user) output file */
#define EX_IOERR    74  /* input/output error */
#define EX_TEMPFAIL 75  /* temp failure; user is invited to retry */
#define EX_PROTOCOL 76  /* remote error in protocol */
#define EX_NOPERM   77  /* permission denied */
#define EX_CONFIG   78  /* configuration error */

#define EX__MAX 78  /* maximum listed value */

#endif /* !_SYSEXITS_H_ */

Code	Error (Karma)	Error (sysexits.h)
1	Actual test failure
64	Invalid cli args	EX_USAGE
78	Fail to parse config file	EX_CONFIG
100	Installation corruption
101	Plugin error (could be almost anything in karma-land if a plugin fails)
101	Failure during startup/teardown
102	Requested a browser that’s not available
103	Unable to start browser
104	Unable to connect to browser
105	Test framework exception (internal test harness error)
106	Fail to parse tested files
107	Disconnect from browser
108	Test timeout
126	Catch All

We should not be using

• 2 and >126 because of http://tldp.org/LDP/abs/html/exitcodes.html
• 64-78 unless it matches to sysexits.h

[dignifiedquire]

@Moumi thanks for the offer but this is quite intricate and will involve a good amount of changes inside the code base so I'm not ready to commit to say we are doing this. Please take a look at the help wanted label for tasks that are ready to be worked on.