Impossible to catch "Error [ERR_STREAM_WRITE_AFTER_END]: write after end"

[AlexandreDecollas]

Hi,

I face an issue when I try to test my nestJs lib resilience.

Here is the process : I have a rest API, calling in fine a service, that calls eventStore.appendEvent. Everything works fine if the connection to the event store is not failing.

For the resilience test, I try to restart my docker container for ES connection, in order to test the reconnection feature of my service and simulate a brutal deconnection.

I then test it with "hey" that jostle the rest service.

My problem is that I have a SIGTERM signal emitted by the grpc lib, and this error :

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at ClientHttp2Stream.Writable.write (internal/streams/writable.js:292:11)
    at /Users/alexandre/workspace/veille/prestashop/nestjs-geteventstore/node_modules/@grpc/grpc-js/src/call-stream.ts:765:26
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
[DEBUG] 11:55:10 Child error
[ERROR] 11:55:10 Error: write after end
[DEBUG] 11:55:10 Disconnecting from child
[DEBUG] 11:55:10 Sending SIGTERM kill to child pid 27627

I did my researches. It seems to happen because of the concurrency, but why a SIGTERM signal? Is it your lib responsibility? Is there a way to catch this very fatal error (the rest is not able to take other request after that)

Do you have any idea about this issue?

Best regards :)

[George-Payne]

Hi Alexandre,

Thanks for the issue.

I was initially thinking this would be fixed by updating the @grpc/grpc-js, they had a related issue around it: https://github.com/grpc/grpc-node/issues/1094, but it didn't fix the problem.

It seems that what is happening is @grpc/grpc-js continues to write the messages we have queued up after the stream has already errored. As the promise has already rejected, these errors are no longer being caught. This results in an unhandled exception, causing node to write the exception to the stderr and exit the process.

I'm fairly sure that grpc-js shouldn't be attempting to write to a closed stream, but I'll investigate it more and open an issue there, if needed.

We can avoid the problem by destroy the stream ourselves, effectively cancelling the messages we have queued, and I've opened a PR to do that: #209

George

[AlexandreDecollas]

Thank you George for your answer,

I by-passed the issue by filtering the uncaught exceptions on process.once directly.

Thank you!