When using pipes other than stdin, stdout, stderr for streaming data into a child process, ENOTCONN error is thrown

ferencbeutel4711 commented 7 years ago

Version:

tested with 7.5 and 8.0

Platform:

OSX Sierra; Darwin Kernel Version 16.3.0 root:xnu-3789.31.2~1/RELEASE_X86_64 x86_64

Example code:

brew install imagemagick

const request = require('request');
const spawn = require('child_process').spawn;

// const subProcess = spawn('convert', ['fd:0', '-size', '100x100', '-'], {stdio: [null, null, null, 'pipe']});
// request.get('https://tinyurl.com/zyksbz7').pipe(subProcess.stdio[0]); // this works

const subProcess = spawn('convert', ['fd:3', '-size', '100x100', '-'], {stdio: [null, null, null, 'pipe']});
request.get('https://tinyurl.com/zyksbz7').pipe(subProcess.stdio[3]); // this does not

subProcess.stdout.on('data', (data) => {
  console.log('new data: ', data);
});

Details:

I am trying to pipe a stream, which i got from request, to the command line in order to manipulate the received data. When i am sending the data to stdin of the child process, everything works fine. In my example, you can see that result by running the commented code rather than the uncommented. If I pipe the stream to a pipe other than stdin, stdout or stderr, which should be possible according to the docs I get the following exception:

events.js:182
      throw er; // Unhandled 'error' event
      ^

Error: shutdown ENOTCONN
    at exports._errnoException (util.js:1026:11)
    at Socket.onSocketFinish (net.js:298:25)
    at emitNone (events.js:105:13)
    at Socket.emit (events.js:207:7)
    at finishMaybe (_stream_writable.js:579:14)
    at endWritable (_stream_writable.js:587:3)
    at Socket.Writable.end (_stream_writable.js:538:5)
    at Socket.end (net.js:490:31)
    at Request.onend (internal/streams/legacy.js:44:10)
    at emitOne (events.js:120:20)
    at Request.emit (events.js:210:7)
    at IncomingMessage.<anonymous> (/Users/ferencbeutel/IdeaProjects/imageprocessing/node_modules/request/request.js:1091:12)
    at Object.onceWrapper (events.js:312:19)
    at emitNone (events.js:110:20)
    at IncomingMessage.emit (events.js:207:7)
    at endReadableNT (_stream_readable.js:1045:12)
    at /Users/ferencbeutel/IdeaProjects/imageprocessing/node_modules/async-listener/glue.js:188:31
    at _combinedTickCallback (internal/process/next_tick.js:102:11)
    at process._tickCallback (internal/process/next_tick.js:161:9)

I tried multiple things, for example passing in the stream object directly when specifying the stdio object (which should be possible too according to the linked doc entry) but then I get a type error, even though I am clearly passing in a stream. I cant see the difference between the two examples I provided and am rather puzzled by the different result.

Thanks for looking into this!

wsxiaoys commented 7 years ago

I met the same issue, after a bit investigation it seems a legacy child process bug on handling any FD >= 3: https://github.com/nodejs/node-v0.x-archive/issues/9419

noahnu commented 7 years ago

Haven't tested with imagemagick, however the following works on CentOS 7:

fd-writer.c

#include <stdio.h>
#include <string.h>

int main(int argc, char *argv[])
{
  if (argc == 1) {
    printf("Usage: %s fd1 [fd2] ...\n", argv[0]);
    return 1;
  }

  char* text = "Hello\n";

  int fd = 1, i = 1;
  while (i < argc && (fd = atoi(argv[i++]))) {
    if (!fd) continue;
    write(fd, text, strlen(text) + 1);
    close(fd);
  }

  return 0;
}

$ gcc ./fd-writer.c -o ./fd-writer

fd-writer will write "Hello" to each file descriptor passed in.

test.js

const { spawn } = require('child_process');

const child = spawn('./fd-writer', [3], {
  stdio: [null, null, null, 'pipe']
});

child.stdio[3].on('data', chunk => {
  process.stdout.write(chunk);
});

$ node test.js

So definitely not an issue with fd >= 3. Have you tried writing directly to the stream? e.g.

subProcess.stdio[3].write(data);

ferencbeutel4711 commented 7 years ago

@noahnu Hi, yes, I've tried that. Writing the data seems fine, however, on closing the input stream (which tells the spawned process that no more data is coming in) I get a similar error. The subprocess I called in this example is imagemagick which supports inputs with fds >= 3, I've tested that aswell.

ahmadsholehin commented 6 years ago

I'm experiencing this as well, under OS X.

matianfu commented 6 years ago

the same error here, node version 8.7, linux

phal0r commented 6 years ago

We can confirm this error under Linux (Debian):

the code runs reliably in windows
it throws the aforementioned error in linux
listening to the error event and ignoring the error leads to successful communication between child and master process
using stdin instead of stdio[3] does not throw the error (the rest of the code is the same)

We used fork to spawn a node js child with ipc as the last parameter

nodejs / node