pg.Client - Cannot read property 'handleRowDescription' of null and Cannot read property 'name' of null

dataform-co / dataform

Dataform is a framework for managing SQL based data operations in BigQuery

https://cloud.google.com/dataform/docs

Apache License 2.0

853 stars 166 forks source link

pg.Client - Cannot read property 'handleRowDescription' of null and Cannot read property 'name' of null #914

Closed lewish closed 4 years ago

lewish commented 4 years ago

Tracking bug, possibly related: https://github.com/brianc/node-postgres/issues/1105, more details to follow.

lewish commented 4 years ago

Still here unfortunately, this time a slightly different format:

TypeError: Cannot read property 'handleCommandComplete' of null
    at Connection.<anonymous> (/app/ts/job-service/job-service-bin.runfiles/npm/node_modules/pg/lib/client.js:346:22)
    at Connection.emit (events.js:210:5)
    at /app/ts/job-service/job-service-bin.runfiles/npm/node_modules/pg/lib/connection.js:109:10
    at Parser.parse (/app/ts/job-service/job-service-bin.runfiles/npm/node_modules/pg-protocol/src/parser.ts:102:9)
    at TLSSocket.<anonymous> (/app/ts/job-service/job-service-bin.runfiles/npm/node_modules/pg-protocol/src/index.ts:7:48)
    at TLSSocket.emit (events.js:210:5)
    at addChunk (_stream_readable.js:308:12)
    at readableAddChunk (_stream_readable.js:289:11)
    at TLSSocket.Readable.push (_stream_readable.js:223:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:182:23)

Still implies the same underlying issue.

BenBirt commented 4 years ago

Right at the time that handleCommandComplete exception was thrown (https://bit.ly/2D4Hm7L), we had loads of pg.Client client errors logged: pg.Client client error 140659317524352:error:14187180:SSL routines:ssl_do_config:bad value:../deps/openssl/openssl/ssl/ssl_mcnf.c:74:section=system_default, cmd=MinProtocol, arg=TLSv1.2 (https://bit.ly/33dtu5B).

BenBirt commented 4 years ago

I have absolutely no idea if it's related, but I have found https://gist.github.com/feymartynov/fdfa1a9691d77f2ef9bd7468ba9b8710, and we do use nodejs' buster image as base.

BenBirt commented 4 years ago

The most recent uncaught exception also had many of these SSL errors just before it. I trawled through runs that were triggered soonish before that happened, and managed to find one customer with an occasional run failure due to these SSL errors: https://app.dataform.co/#/5402252606963712/runlist

Current hypothesis:

pg.Client encounters an SSL error (for some reason, no idea why), reports it to the error handler
that pg.Client is then reused for a later query, but is in an incorrect state and pg cannot correctly run queries using it - and throws the uncaught exceptions that we see

lewish commented 4 years ago

Haven't seen this for a week now