Force pipeline termination from pipeline manager

vmware-archive / database-stream-processor

Streaming and Incremental Computation Framework

Other

225 stars 20 forks source link

Force pipeline termination from pipeline manager #397

Open ryzhyk opened 1 year ago

ryzhyk commented 1 year ago

The runner handles the /shutdown command by first calling controller.stop() and then sending a termination message to the actix dispatch loop. Both operations can be slow or even get stuck in some cases. We will need a way to enforce eventual termination from the pipeline manager by killing the process/pod after a timeout.

@lalithsuresh

lalithsuresh commented 1 year ago

@ryzhyk I get the operations being slow, but do we know when/why they get stuck? Because that would be a bug.

ryzhyk commented 1 year ago

controller.stop() waits for any in-progress iteration of the circuit to complete. Which should normally be fast, but can take a while if there's a perf issue in the circuit or we're processing a very large input batch. Another possible reason is buggy connectors that refuse to shutdown instantly or at all.
actix is not supposed to take long, but I think I've seen it take a while to shutdown. Not sure what's up with that.