Closed lewish closed 4 years ago
Still here unfortunately, this time a slightly different format:
TypeError: Cannot read property 'handleCommandComplete' of null
at Connection.<anonymous> (/app/ts/job-service/job-service-bin.runfiles/npm/node_modules/pg/lib/client.js:346:22)
at Connection.emit (events.js:210:5)
at /app/ts/job-service/job-service-bin.runfiles/npm/node_modules/pg/lib/connection.js:109:10
at Parser.parse (/app/ts/job-service/job-service-bin.runfiles/npm/node_modules/pg-protocol/src/parser.ts:102:9)
at TLSSocket.<anonymous> (/app/ts/job-service/job-service-bin.runfiles/npm/node_modules/pg-protocol/src/index.ts:7:48)
at TLSSocket.emit (events.js:210:5)
at addChunk (_stream_readable.js:308:12)
at readableAddChunk (_stream_readable.js:289:11)
at TLSSocket.Readable.push (_stream_readable.js:223:10)
at TLSWrap.onStreamRead (internal/stream_base_commons.js:182:23)
Still implies the same underlying issue.
Right at the time that handleCommandComplete
exception was thrown (https://bit.ly/2D4Hm7L), we had loads of pg.Client client errors
logged: pg.Client client error 140659317524352:error:14187180:SSL routines:ssl_do_config:bad value:../deps/openssl/openssl/ssl/ssl_mcnf.c:74:section=system_default, cmd=MinProtocol, arg=TLSv1.2
(https://bit.ly/33dtu5B).
I have absolutely no idea if it's related, but I have found https://gist.github.com/feymartynov/fdfa1a9691d77f2ef9bd7468ba9b8710, and we do use nodejs' buster
image as base.
The most recent uncaught exception also had many of these SSL errors just before it. I trawled through runs that were triggered soonish before that happened, and managed to find one customer with an occasional run failure due to these SSL errors: https://app.dataform.co/#/5402252606963712/runlist
Current hypothesis:
pg.Client
encounters an SSL error (for some reason, no idea why), reports it to the error
handlerpg.Client
is then reused for a later query, but is in an incorrect state and pg
cannot correctly run queries using it - and throws the uncaught exceptions that we seeHaven't seen this for a week now
Tracking bug, possibly related: https://github.com/brianc/node-postgres/issues/1105, more details to follow.