Server blocks when connection is closed before end of stream

[kosniaz]

Server doesn't close the connection when kaldigrpc_client disconnects before the end of stream. This has been noted both in the programmatic use, when we run the ILSPASRClient.streaming_recognize method, as well as when running kaldigrpc-transcribe --streaming from the cli.

Expected Behavior

Streaming recognition works, and when we stop streaming audio and break the client print loop, the server worker disconnects from the grpc channel and is available again.

Current Behavior

Streaming recognition still works, but when we stop streaming audio and break the client print loop, the server worker doesn't disconnect from the grpc channel, and stays blocked for at least 30 minutes.

Possible Solution

Close the connection on the server side too. Possibly redefine the meaning of "end_of_stream"?

Steps to Reproduce

1. Run a kaldigrpc-server container, and note the number of max_workers, e.g. 5
2. Run a streaming recognition task and cancel it while it is being executed. E.g. on chomsky, run kaldigrpc-transcribe --streaming --port 1234 geopar.wav and then Ctrl-C it after some seconds
3. Repeat above step as many times as the number of max workers plus 1, e.g. 5+1
4. Note that the last run doesn't connect to the server at all.

Context (Environment)

In the context of Virtual Assistants, where we need to perform an action when we recognize certain commands, we should be able to close the grpc channel abruptly.

Looking forwrard to further discussion and will be happy to assist in the implementation of a solution!

[kosniaz]

Also, it could be useful to add a deadline to the connection. https://grpc.io/blog/deadlines/

[kosniaz]

I will try again with the env variable GRPC_VERBOSITY=DEBUG

[georgepar]

TODO: Add verbose logs / error handling / connection deadline

[georgepar]

Fixed: https://github.com/georgepar/kaldi-grpc-server/pull/13