Setup correct cleanup of auxiliary threads in executors

[tcompa]

The minimal example below shows that the __exit__ method is not called if an error is raised in __init__, and it shows a possibly pattern for always cleaning up.

class ContextManager():

    def __init__(self, raise_error: bool):
        try:
            print("init")
            if raise_error:
                raise RuntimeError()
        except Exception as e:
            self.cleanup()
            raise e

    def __enter__(self):
        print("enter")

    def cleanup(self):
        print("cleanup")

    def __exit__(self, a, b, c):
        self.cleanup()
        print("exit", a, b, c)

with ContextManager(raise_error=False):
    print("operations")
print()

with ContextManager(raise_error=True):
    print("operations")
print()

output

init
enter
operations
cleanup
exit None None None

init
cleanup
Traceback (most recent call last):
  File "/tmp/x.py", line 29, in <module>
    with ContextManager(raise_error=True):
  File "/tmp/x.py", line 12, in __init__
    raise e
  File "/tmp/x.py", line 9, in __init__
    raise RuntimeError()
RuntimeError

[tcompa]

Changes coming with https://github.com/fractal-analytics-platform/fractal-server/pull/1793:

• Some tests were using SLURM Executor objects directly rather than as context manager (thus skipping the __exit__ cleanup)
• Some tests were triggering exceptions from within Executor __init__ methods, which turns out to not perform the expected cleanup. We manually added the thread cleanup to the specific errors.
• To improve our detection of spurious threads that remain open, we added a fixture that counts threads before/after each tests and raises an error if appropriate. The specific case we had to treat was a paramiko SSH connection leaving an open thread. It's still TBD whether this is a CI issue or an issue with the underlying PR (#1790)

[tcompa]

Closed with #1790. A different issue is due, re: whether failed paramiko connections leave a trace of spurious threads open - cc @mfranzon