f5devcentral / grpc-js-0.2-modified

grpc-js , version 0.2 (old version) with some modifications to work with node 8.11.1
0 stars 0 forks source link

uncaughtException: this.http2Stream.close is not a function #1

Open garrettdieckmann opened 3 years ago

garrettdieckmann commented 3 years ago

When using this library in F5 Telemetry Streaming, we observe frequent restarts of the restnoded process with the following in the log file:

Wed, 03 Mar 2021 18:25:13 GMT - error: uncaughtException: this.http2Stream.close is not a function date=Wed Mar 03 2021 18:25:13 GMT+0000 (UTC), pid=25114, uid=198, gid=198, cwd=/var/service/restnoded, execPath=/usr/bin/f5-rest-node, version=v8.11.1, argv=[/usr/bin/f5-rest-node, /usr/share/rest/node/src/restnode.js, -p, 8105, --logLevel, finest, -i, /var/log/restnoded/restnoded.log, -s, none], rss=753487872, heapTotal=569597952, heapUsed=528157592, external=127697705, loadavg=[0.33642578125, 0.5595703125, 0.87939453125], uptime=551809
TypeError: this.http2Stream.close is not a function
    at Http2CallStream.destroyHttp2Stream (/var/config/rest/iapps/f5-telemetry/node_modules/grpc-js-0.2-modified/build/src/call-stream.js:326:30)
    at Http2CallStream.cancelWithStatus (/var/config/rest/iapps/f5-telemetry/node_modules/grpc-js-0.2-modified/build/src/call-stream.js:330:14)
    at Http2CallStream.call.on (/var/config/rest/iapps/f5-telemetry/node_modules/grpc-js-0.2-modified/build/src/client.js:70:22)
    at emitNone (events.js:111:20)
    at Http2CallStream.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
Wed, 03 Mar 2021 18:25:13 GMT - info: uncaught exception
Wed, 03 Mar 2021 18:25:16 GMT - info: process exit

This looks similar to the work that was done when forking this library, of replacing the .close() functions: https://github.com/f5devcentral/grpc-js-0.2-modified#this-is-a-modified-version-of-grpc-js-version-02

Expected behavior: Exception is caught and handled, or the function call is updated so that it does not call a non-existent function for NodeJS v8.11.1.

kaustriaf5 commented 3 years ago

From trying to troubleshoot on a repro box, this seems to be triggered from an attempt to destroy the stream after server returns a null response:

https://github.com/f5devcentral/grpc-js-0.2-modified/blob/master/build/src/client.js#L70

Which calls cancelWithStatus https://github.com/f5devcentral/grpc-js-0.2-modified/blob/master/build/src/call-stream.js#L329

Which then calls https://github.com/f5devcentral/grpc-js-0.2-modified/blob/master/build/src/call-stream.js#L320

(Trace from a modified src on repro box)

 Err on stream.close:"Not enough responses received"
    at Http2CallStream.destroyHttp2Stream (..../node_modules/grpc-js-0.2-modified/build/src/call-stream.js:329:19)
    at Http2CallStream.cancelWithStatus (...../node_modules/grpc-js-0.2-modified/build/src/call-stream.js:334:14)
    at Http2CallStream.call.on (...../node_modules/grpc-js-0.2-modified/build/src/client.js:70:22)
    at emitNone (events.js:111:20)
    at Http2CallStream.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

Looks like http2Stream is not correctly being dereferenced/set to null in this instance, so it passes the check and close attempt is made:

    destroyHttp2Stream() {
        // The http2 stream could already have been destroyed if cancelWithStatus
        // is called in response to an internal http2 error.
        if (this.http2Stream !== null && !this.http2Stream.destroyed) {
            /* TODO(murgatroid99): Determine if we want to send different RST_STREAM
             * codes based on the status code */
            this.http2Stream.close(NGHTTP2_CANCEL);
        }
    }