Ability to close Workers & schedulers properly

[brian-mulier-p]

Expected Behavior

We should be able to try-with-resource around a worker and a scheduler in tests for eg. (but also ensure proper shutdown of the app).

Actual Behaviour

Closing a worker will close all its related queues. Also, closing 1 queue leads to every queue unable to execute any task anymore (since queue-closing will close the executor pool which is currently static and with all queues).

Steps To Reproduce

No response

Environment Information

• Kestra Version:
• Operating System (OS / Docker / Kubernetes):
• Java Version (If not docker):

Example flow

No response

[brian-mulier-p]

As discussed, the proper way to handle it would probably be to hold the Runnable to stop queue consumptions in a common util

[loicmathieu]

Another way would be to let Micronaut close queues when closing the application context using bean lifecycle (which handle beans dependencies so should be safe).

[brian-mulier-p]

As discussed again with Loic, the question is, do we want to be able to try-with properly around a scheduler & a worker or forcing a Micronaut context reconstruct is enough ? As said the Micronaut context based solution would add some more test time as we would need to force the deletion of the context after each test which is using an embed Scheduler / worker. However it leads to better time-to-delivery and probably side-effect free.

[loicmathieu]

We can do both by providing adding @PreDestroy on the close method. The important point is that we need to remove all direct calls to close, except on test.

[brian-mulier-p]

Yes but then we still have to remove the queues closing calls if we want to be able to close the worker & scheduler in tests

[loicmathieu]

@brian-mulier-p can we condidere that this has been done with the new way we close our server component with the changes done in 0.16 by @fhussonnois ?

[brian-mulier-p]

Idk we have to test if the try with worker / scheduler works properly