Collect tests results even when some tests fail (or timeout)

closes #374

Collect tests results even when some tests fail (or timeout)

Tricky case: when tests failed, some tests may have not been executed, so tests results files may be missing. We still want to collect them if possible, without raising extra error if we cant. We still want to fail when the collection failed when the tests succeeded though.

Current solution: duplicate the tests results collections commands, wrap tests, and individual collections, with try/catch/rethrow and help messages.

On bash, use https://stackoverflow.com/a/25180186/15151442 for try/catch.

Maybe a better way to handle that would be to have a "super cmd" that takes the individual test results collections cmds, and handle the error handling and duplication in generate_command_bash|pipeline()?

Also:

always store container id in txt files, even if tests failed
always late-collect cobertura reports. We reach that point either via normal flow where we do want an error if no file found, or via test error where we don't want an error if no file found. Luckily, inline part of cobertura will already have failed if file is missing; with proper error handling: we can ignore all that here.

To test things, uncomment #- lets_fail_here in testweb/dmake.yml

Results

https://build.deepomatic.com/job/dmake/job/PR-482/20 Here is how it looks like when a test fails, with remaining tests results to collect (and DMAKE_DEBUG=1, it's less verbose in real life): in classic logs: Failed in branch test @ dmake-test/test-web

in blue ocean: blue ocean graph

blue ocean steps logs

Also

Jenkins collect dmake self tests results even on error
Add color to dmake self tests output
quickfix escaping echo messages containing ; We probably need more escaping: bash escaping, but the generate_bash echo escaping is also wrong (it uses groovy string escaping, not bash escaping...), so it's even between bash and jenkinsfile generation.
Add indentation to bash generated DMakefile, like it's already done for Jenkinsfile

Deepomatic / dmake