Stream pipe fails in Node 10.15.3 when piping file with data

[amber-cd]

We recently upgraded from an older version of node to Node 10, and ever since I've been experiencing an issue with a read stream from a remote server just not working when attempting to pipe from the SFTP read stream.

An example of our setup:

console.log('Creating the stream');
let streamPromise = new Promise((resolve, reject) => {
    sftp.createReadStream('/Mydata/download_report.csv').then(stream => {
        console.log('Read stream created! Creating the write stream');
        let writeStream = fs.createWriteStream('download_report.csv');
        console.log('Write stream created!');
        stream.on('error', (error) => {
            console.error('Error in read stream', error);
            return reject(error);
        });
        stream.on('data', (data) => {
            console.log('data in read stream', Buffer.from(data, 'base64').toString());
        });
        stream.on('end', () => {
            try {
                writeStream.end();
              } catch(err) {
                // Suppress warning, but still resolve
                return resolve('Done! Resolved from stream.on end in the error handler for writeStream.end()');
              }
        });
        writeStream.on('data', (data) => {
            console.log('data in write stream', Buffer.from(data, 'base64').toString());
        });
        writeStream.on('error', error => {
            console.error('Error in write stream', error);
            return reject(error);
        });
        writeStream.on('finish', () => {
            console.log('Finish called in writeStream.')
            return resolve('Done! Resolved from stream.on finish in writeStream.');
        });

        console.log('Piping the stream');
        stream.pipe(writeStream);
    }).catch(error => {
        console.error('Error creating read stream', error);
        return reject(error);
    });
});

streamPromise.then((results) => {
    console.log('Stream Promise resolved!');
    console.log(results);
}).catch(console.error);

In our old Node installation, this logs out fine:

Creating the stream
Read stream created! Creating the write stream
Write stream created!
Piping the stream
data in read stream
Col A,Col B,Col C,Col D
a,b,c,d
Finish called in writeStream.
Done! Resolved from stream.on finish in writeStream.

Checking the local FS shows the file with all the expected data.

But in Node 10, the log is just:

Creating the stream
Read stream created! Creating the write stream
Write stream created!
Piping the stream

Then... nothing else. It just stops, and the local file is empty.

[brokenbot]

Hmm... I haven't tested in node 10 yet. I still need to finish fixing the odd behavior when listing files with relatives paths, so I'll add node 10 support along with it. It'll be a few days before I have time to tackle it but look into why the stream isn't piping through.

[brokenbot]

Sorry for the delay in testing this.

Just tested in node 10.15.3 and a bunch of other versions. I'm not exactly sure what gives but calling stream.pipe() should resume the stream and start the transfer. However starting in node 10.0.0 this longer works as expected, but calling stream.read() after stream.pipe() resolve the issue.

I'm not sure if this a bug in node or the underlying ssh2-streams package, or by design some how.

let me know that solve your problem.

Thanks,

Dave

[amber-cd]

Hrm.

Both

stream.pipe(writeStream);
writeStream.write(stream.read());

stream.pipe(writeStream);
stream.read();

do indeed seem to work around this issue. It's a bit unintuitive, and I'm running into a separate issue with GCS, but I think it should work for these purposes. Thank you!