"Arbitrary" test failures after previous test run have been aborted

[Nicoretti]

Attention:

• Failures due to this bug can look "arbitrary" or "sporadic" depending on how and when the pre conditions for this bug have been triggered.
• Not all possible states / scenarios of this bug have been observed or analyzed yet, so this description may only contain a subset of the actual behavior(s).

Steps to reproduce:

1. Start all tests via scripts/test/run_all_tests.sh
2. Wait until at least 1-2 docker processes are spawned/running + DB container (docker ps)
3. Stop/Interrupt the test execution using STRG + C (SIGINT)
4. As of now, it looks like when the DB docker container is left behind, one will get some sort of errors in successive test executions until the "containers" are cleaned up.
5. Execute all test (some tests should fail until the containers are cleaned up)
   • docker rm <hash> ...
   • docker image prune

Root Cause: As of now, it looks like the left behind Exasol DB container causes test failures in successive test executions.

Possible strategies to address this bug:

• Install a signal handler which will make sure all containers are cleaned up in case of a SIGINT abort
• Add a warning if test(s) are started while "orphaned" container(s) are around (especially the DB container)
• Add a signal handler that will print a warning if the process is aborted using STRG + C
• Document the issue and its counter measures Known Issues

Potentially related issues:

• https://github.com/exasol/integration-test-docker-environment/issues/109
• https://github.com/exasol/integration-test-docker-environment/issues/24

[tkilias]

@Nicoretti We actually have cleanup code in place on multiple levels which reacts to SIGINT. The tasks which are starting the DB, remove it when one of them fail and the Test abstraction cleans up as well and the teardowns, too. Can you confirm it was a single CTRL+C, because I so far only saw leftovers after a double CTRL+C which is more or less a SIGKILL. This can also happen easily, because the cleanup takes a while.

[Nicoretti]

@tkilias I am quite confident I pressed it at least twice. So I don't think it is a bug in this case. Still I don't think pressing 2x ctrl + c ending up being a "SIGKILL" is obvious to an end user. We may should add a warning regarding this.

[Nicoretti]

@tkilias just validated it, 1x ctrl + c works well but 2x ctrl + c may leave leftovers behind. So I would say it isn't a bug.

Current Implementation:

• On the first ctrl + c the end user gets informed that the "program" is shutting down
• On the second ctrl + c the end user gets the message Aborted!

To sum it up: It works as expected, but the implication(s) of the Aborted! could be made more clear.

[tkilias]

@tkilias just validated it, 1x ctrl + c works well but 2x ctrl + c may leave leftovers behind. So I would say it isn't a bug.

Current Implementation:

* On the first `ctrl + c` the end user gets informed that the "program" is shutting down

* On the second `ctrl + c` the end user gets the message `Aborted!`

To sum it up: It works as expected, but the implication(s) of the Aborted! could be made more clear.

@Nicoretti Hm, interesting, I am actually not sure, where the message Aborted! comes from.