search

googleapis / google-cloud-node

Google Cloud Client Library for Node.js
https://cloud.google.com/nodejs
Apache License 2.0
2.92k stars 594 forks source link

Getting ESOCKETTIMEOUT when streaming through file.createReadStream() #2317

Closed MuthuD closed 7 years ago

MuthuD commented 7 years ago

Hi,

We transform a large binary file (>150MB up to GBs) and publish the data to pubsub.

OS: Google App Engine (Assuming its windows)
Node.js version: LTS
npm version: LTS
google-cloud-node version: HEAD

Steps to reproduce

  1. require google-cloud
  2. let file = bucket.file([largebinaryfile]);
  3. let readStream = file.createReadStream();
  4. file.pipe(transformStream).pipe(writeStream)

readStream: Reads large file from bucket directly from bucket transformStream: Does some operation writeStream: publishes transformed data to cloud pubsub

Problem, readStream receives ESOCKETTIMEDOUT error more often, how to handle this is this value configurable through code?

Thanks

stephenplusplus commented 7 years ago

Can you double check the version of google-cloud-node that you're using? Only the latest release will retry on errors such as a socket timeout.

MuthuD commented 7 years ago

Hi Stephen,

Thanks for the response.

I'm using version @google-cloud/storage: "0.7.0"

I will try with version 1.1.1 and let you know if there is any issue.

Thanks

stephenplusplus commented 7 years ago

@MuthuD how did it go?

Scarysize commented 6 years ago

Hey there, thought I would chime in here before opening a new issue in the nodejs-storage repo. I got the exact same issue on version 2.0.1.

Here is the code producing the error:

const readline = require('readline');

const storage = new Storage({projectId: config.projectId});
const bucket = storage.bucket(config.bucketName);
const remoteFile = bucket.file('events.txt');

const lineReader = readline.createInterface(remoteFile.createReadStream());

lineReader.on('line', /* do stuff */);

The code runs on GCE:

It reliably times out after 60 seconds of streaming.

Let me know if you need any further information.

JustinBeckwith commented 6 years ago

Thanks! Opening this on the nodejs-storage repo is absolutely the way to go though 👍

Scarysize commented 6 years ago

👍 Okay, added a comment to a similar issue: https://github.com/googleapis/nodejs-storage/issues/27#issuecomment-420534576