search

apla / node-clickhouse

Yandex ClickHouse driver for nodejs
MIT License
216 stars 50 forks source link

Error: socket hang up #8

Closed jianbingfang closed 4 years ago

jianbingfang commented 6 years ago

When I try to insert a very big file (23 GB) using , it always return a error:

{ Error: socket hang up
    at createHangUpError (_http_client.js:331:15)
    at Socket.socketOnEnd (_http_client.js:423:23)
    at emitNone (events.js:111:20)
    at Socket.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9) code: 'ECONNRESET' }

My insert code:

var tsvStream = fs.createReadStream ('data.tsv');  // file size of data.tsv is 23 GB
var clickhouseStream = clickHouse.query ('INSERT INTO test_table', {inputFormat: 'TSV'});
tsvStream.pipe (clickhouseStream);

How to resolve this issue? And I wonder is there throttling in streaming insert?

nezed commented 5 years ago

Could you please provide more info about data structure that you've uploading and your setup/environment?

This problem looks like network failure or DBMS crash.

Also, there is not throttling in nodejs streams, but you can made it yourself, e.g. by implementing throttling while reading file.