containerd / ttrpc-rust

Rust implementation of ttrpc (GRPC for low-memory environments)
Apache License 2.0
195 stars 45 forks source link

send all pending responses before shutting down #228

Closed alex-matei closed 4 months ago

alex-matei commented 4 months ago

The async implementation doesn't wait for responses to be sent to clients when shutting down. These responses are lost if the application exists fast enough before the write task gets to send them. That means that the response for a request that triggers shutdown of the server might be lost without the client knowing that the server was actually stopped.

I observed this behavior in kata-containers. The kata shim sends DestroySandboxRequest request to kata-agent, which runs inside the VM, to tell it to clean up and exit. The handler code for this request sets an event to tell the main thread that the process can be stopped. After that it returns an empty response to the shim. The main thread calls ttrpc_server_obj.shutdown() and the process exits. The ttrpc shutdown code doesn't wait for the reply to be sent. As such, kata shim might receive an error in the middle of the rpc call to DestroySandbox without the call actually failing at all.