Closed pushred closed 10 years ago
The warning is "normal" and simply means you're processing more than 10 files in parallel. gulp-run spawns a new process to handle every file it receives as input and processes them in parallel. For each file, a logger object is created to handle piping the stdout of the child process to the stdout of the main process. Piping involves creating event listeners on process.stdout
, hence the warning.
This log system could definitely be refactored to avoid this, but no memory is actually being leaked (as of 93030182533a5abbc6a1621849c8a35cf0d34583), so you can safely increase the listener count on process.stdout
to remove the warning.
Consider the following two examples that run test a = a
one-hundred times.
var count = 100;
function almostDone() {
count -= 1;
if (count === 0) done();
}
for (var i = count; i > 0; i -= 1) {
run('test a = a').exec(almostDone);
}
Since exec
is asynchronous, this example spawns 100 processes in parallel and calls done
when they have all exited. You'll see the memory leak warning here because all 100 of the processes attach listeners to process.stdout
. Since gulp pipelines are asynchronous, this is what happens and is usually what should happen from a performance standpoint.
var count = 100;
function test() {
run('test a = a').exec(function () {
count -= 1;
if (count > 0) test();
else done();
});
}
test();
This example also spawns 100 processes, but it waits for the previous to finish before spawning the next. In this case, you won't see the warning because each process releases it's listeners before the next process starts. And since the warning is not thrown, this example shows that each execution properly cleans up after itself.
As far as I can tell, the solution is process.stdout.setMaxListeners(n + 1)
where n
is the number of files being processed. Or just process.stdout.setMaxListeners(Infinity)
.
Awesome, thanks so much for this detailed explanation. This all makes perfect sense. The concurrency is definitely desirable!
I'm using gulp-run in conjunction with gulp-tap to pass the paths of files I'm watching on to a
theme upload
command in shopify_theme Ruby gem. Generally this works but I've started seeing the following warning as the number of files in my project increases. The uploads are still successful, but this is making for some noisy log output.(example gulp-run log output)
Not sure if this is related to https://github.com/gulpjs/gulp/issues/432 but seems to be coming from
Transform.commandStream.exec
in gulp-run. I'm running 0.10.28.