Flaky test: tests/test_service_cli.py::test_shutdown_called_on_shutdown_signal[Signals.SIGINT]

elastic / connectors

Source code for all Elastic connectors, developed by the Search team at Elastic, and home of our Python connector development framework

https://www.elastic.co/guide/en/enterprise-search/master/index.html

Other

71 stars 126 forks source link

Flaky test: tests/test_service_cli.py::test_shutdown_called_on_shutdown_signal[Signals.SIGINT] #2517

Open artem-shelkovnikov opened 4 months ago

artem-shelkovnikov commented 4 months ago

One unit test is flaky and from time to time takes too much time to run, causing CI to terminate the process: bin/pytest tests/test_service_cli.py::test_shutdown_called_on_shutdown_signal

artem-shelkovnikov commented 4 months ago

Tried looking into it and it's really weird, feels like it's an interrupt that happens in a wrong moment.

Testing it a bit locally showed that the test does not interrupt because some services (in my case it was job cleanup service) did not exit.

I think it happens because SIGINT/SIGTERM happens in between service startup - e.g. Job Scheduling/Job Execution Services has just started, next Job Cleanup Service will attempt to start, but interrupt happens before and service just starts.

☝️ This info is just a theory, I was unable to prove or discard it.

seanstory commented 3 months ago

I wonder if this is related to https://github.com/elastic/connectors/issues/2568

artem-shelkovnikov commented 3 months ago

@seanstory I have a feeling it's the same issue.

It feels like CTRL+C in some particular place of code does not correctly trigger the cleanup and after cleanup was started some couroutines were still executed.