Compatibility with gulp 5?

[marekdedic]

Hi, I recently tried to update to gulp v5 across multiple repositories using merge-stream. In most of them, it worked just fine, but I am hitting an issue in scout-handbook/odymaterialy.skauting.cz#12. When I do e.g. gulp copy:local, it runs the following code:

return merge(
    gulp.src(['src/api-config.php', 'src/api-secrets.php', 'src/client-config.json', 'src/google8cbe14e41a3d2e27.html', 'src/pgp-key.asc', 'src/privacy.html', 'src/robots.txt'])
        .pipe(gulp.dest('dist/')),
    gulp.src(['src/assetlinks.json', 'src/security.txt'])
        .pipe(gulp.dest('dist/.well-known/')),
    );

And I get:

node:events:496
      throw er; // Unhandled 'error' event
      ^

<ref *2> Error: Writable stream closed prematurely
    at Pipeline.done (/home/user/odymaterialy.skauting.cz/node_modules/streamx/index.js:444:43)
    at PassThrough.emit (node:events:530:35)
    at PassThrough.emit (node:domain:551:15)
    at emitCloseNT (node:internal/streams/destroy:147:10)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)
Emitted 'error' event on Domain instance at:
    at Transform.emit (node:domain:539:12)
    at ReadableState.afterDestroy (/home/user/odymaterialy.skauting.cz/node_modules/streamx/index.js:493:19)
    at Transform._destroy (/home/user/odymaterialy.skauting.cz/node_modules/streamx/index.js:629:5)
    at ReadableState.updateNonPrimary (/home/user/odymaterialy.skauting.cz/node_modules/streamx/index.js:386:16)
    at ReadableState.update (/home/user/odymaterialy.skauting.cz/node_modules/streamx/index.js:367:71)
    at ReadableState.updateReadNT (/home/user/odymaterialy.skauting.cz/node_modules/streamx/index.js:536:10)
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
  domainEmitter: <ref *1> Transform {
    _events: [Object: null prototype] {
      newListener: [Function: newListener],
      pipe: [Function: bound onceWrapper] { listener: [Function: onPipe] }
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    _duplexState: 459316,
    _readableState: ReadableState {
      stream: [Circular *1],
      queue: FastFIFO {
        hwm: 16,
        head: FixedFIFO {
          buffer: [ undefined, undefined, <14 empty items> ],
          mask: 15,
          top: 2,
          btm: 2,
          next: null
        },
        tail: FixedFIFO {
          buffer: [ undefined, undefined, <14 empty items> ],
          mask: 15,
          top: 2,
          btm: 2,
          next: null
        },
        length: 0
      },
      highWaterMark: 0,
      buffered: 0,
      readAhead: true,
      error: [Circular *2],
      pipeline: Pipeline {
        from: [Circular *1],
        to: null,
        afterPipe: null,
        error: [Circular *2],
        pipeToFinished: false
      },
      byteLength: [Function: defaultByteLength],
      map: null,
      pipeTo: <ref *3> Composer {
        _events: [Object: null prototype] {
          newListener: [ [Function: newListener], [Function: removeSink] ],
          removeListener: [Function: removeSink],
          piping: [Function: markSink],
          finish: [Function: bound finished],
          drain: [Function: bound afterDrain],
          end: [Function: bound onceWrapper] {
            listener: [Function: bound remove]
          }
        },
        _eventsCount: 6,
        _maxListeners: undefined,
        _duplexState: 17236508,
        _readableState: ReadableState {
          stream: [Circular *3],
          queue: FastFIFO {
            hwm: 16,
            head: [FixedFIFO],
            tail: [FixedFIFO],
            length: 0
          },
          highWaterMark: 0,
          buffered: 0,
          readAhead: true,
          error: [Circular *2],
          pipeline: Pipeline {
            from: null,
            to: null,
            afterPipe: null,
            error: [Circular *2],
            pipeToFinished: true
          },
          byteLength: [Function: defaultByteLength],
          map: null,
          pipeTo: PassThrough {
            _events: [Object],
            _readableState: [ReadableState],
            _writableState: [WritableState],
            allowHalfOpen: true,
            _maxListeners: 0,
            _eventsCount: 7,
            add: [Function: add],
            isEmpty: [Function: isEmpty],
            [Symbol(shapeMode)]: true,
            [Symbol(kCapture)]: false,
            [Symbol(kCallback)]: null
          },
          afterRead: [Function: bound afterRead],
          afterUpdateNextTick: [Function: bound updateReadNT]
        },
        _writableState: WritableState {
          stream: [Circular *3],
          queue: FastFIFO {
            hwm: 16,
            head: [FixedFIFO],
            tail: [FixedFIFO],
            length: 0
          },
          highWaterMark: 0,
          buffered: 0,
          error: [Circular *2],
          pipeline: Pipeline {
            from: [Circular *1],
            to: null,
            afterPipe: null,
            error: [Circular *2],
            pipeToFinished: false
          },
          drains: null,
          byteLength: [Function: defaultByteLength],
          map: null,
          afterWrite: [Function: bound afterWrite],
          afterUpdateNextTick: [Function: bound updateWriteNT]
        },
        _writable: <ref *4> Transform {
          _events: [Object: null prototype] {
            newListener: [Function: newListener],
            drain: [Function: bound _continueWrite],
            finish: [Function: bound _maybeFinal],
            error: [Array],
            close: [Function: bound destroy]
          },
          _eventsCount: 5,
          _maxListeners: undefined,
          _duplexState: 17236508,
          _readableState: ReadableState {
            stream: [Circular *4],
            queue: [FastFIFO],
            highWaterMark: 0,
            buffered: 0,
            readAhead: true,
            error: Error: Stream was destroyed
                at Object.<anonymous> (/home/user/odymaterialy.skauting.cz/node_modules/streamx/index.js:2:26)
                at Module._compile (node:internal/modules/cjs/loader:1369:14)
                at Module._extensions..js (node:internal/modules/cjs/loader:1427:10)
                at Module.load (node:internal/modules/cjs/loader:1206:32)
                at Module._load (node:internal/modules/cjs/loader:1022:12)
                at Module.require (node:internal/modules/cjs/loader:1231:19)
                at require (node:internal/modules/helpers:179:18)
                at Object.<anonymous> (/home/user/odymaterialy.skauting.cz/node_modules/glob-stream/index.js:9:16)
                at Module._compile (node:internal/modules/cjs/loader:1369:14)
                at Module._extensions..js (node:internal/modules/cjs/loader:1427:10),
            pipeline: [Pipeline],
            byteLength: [Function: defaultByteLength],
            map: null,
            pipeTo: [Transform],
            afterRead: [Function: bound afterRead],
            afterUpdateNextTick: [Function: bound updateReadNT]
          },
          _writableState: WritableState {
            stream: [Circular *4],
            queue: [FastFIFO],
            highWaterMark: 0,
            buffered: 0,
            error: Error: Stream was destroyed
                at Object.<anonymous> (/home/user/odymaterialy.skauting.cz/node_modules/streamx/index.js:2:26)
                at Module._compile (node:internal/modules/cjs/loader:1369:14)
                at Module._extensions..js (node:internal/modules/cjs/loader:1427:10)
                at Module.load (node:internal/modules/cjs/loader:1206:32)
                at Module._load (node:internal/modules/cjs/loader:1022:12)
                at Module.require (node:internal/modules/cjs/loader:1231:19)
                at require (node:internal/modules/helpers:179:18)
                at Object.<anonymous> (/home/user/odymaterialy.skauting.cz/node_modules/glob-stream/index.js:9:16)
                at Module._compile (node:internal/modules/cjs/loader:1369:14)
                at Module._extensions..js (node:internal/modules/cjs/loader:1427:10),
            pipeline: null,
            drains: null,
            byteLength: [Function: defaultByteLength],
            map: null,
            afterWrite: [Function: bound afterWrite],
            afterUpdateNextTick: [Function: bound updateWriteNT]
          },
          _transformState: TransformState {
            data: null,
            afterTransform: [Function: bound afterTransform],
            afterFinal: null
          },
          _transform: [Function: normalize],
          [Symbol(shapeMode)]: false,
          [Symbol(kCapture)]: false
        },
        _readable: <ref *5> Transform {
          _events: [Object: null prototype] {
            newListener: [Function: newListener],
            data: [Function: bound _pushData],
            end: [Function: bound _pushEnd],
            error: [Array],
            close: [Array],
            finish: [Array],
            drain: [Function: bound afterDrain]
          },
          _eventsCount: 7,
          _maxListeners: undefined,
          _duplexState: 17238300,
          _readableState: ReadableState {
            stream: [Circular *5],
            queue: [FastFIFO],
            highWaterMark: 0,
            buffered: 0,
            readAhead: true,
            error: Error: Stream was destroyed
                at Object.<anonymous> (/home/user/odymaterialy.skauting.cz/node_modules/streamx/index.js:2:26)
                at Module._compile (node:internal/modules/cjs/loader:1369:14)
                at Module._extensions..js (node:internal/modules/cjs/loader:1427:10)
                at Module.load (node:internal/modules/cjs/loader:1206:32)
                at Module._load (node:internal/modules/cjs/loader:1022:12)
                at Module.require (node:internal/modules/cjs/loader:1231:19)
                at require (node:internal/modules/helpers:179:18)
                at Object.<anonymous> (/home/user/odymaterialy.skauting.cz/node_modules/glob-stream/index.js:9:16)
                at Module._compile (node:internal/modules/cjs/loader:1369:14)
                at Module._extensions..js (node:internal/modules/cjs/loader:1427:10),
            pipeline: null,
            byteLength: [Function: defaultByteLength],
            map: null,
            pipeTo: null,
            afterRead: [Function: bound afterRead],
            afterUpdateNextTick: [Function: bound updateReadNT]
          },
          _writableState: WritableState {
            stream: [Circular *5],
            queue: [FastFIFO],
            highWaterMark: 0,
            buffered: 0,
            error: Error: Stream was destroyed
                at Object.<anonymous> (/home/user/odymaterialy.skauting.cz/node_modules/streamx/index.js:2:26)
                at Module._compile (node:internal/modules/cjs/loader:1369:14)
                at Module._extensions..js (node:internal/modules/cjs/loader:1427:10)
                at Module.load (node:internal/modules/cjs/loader:1206:32)
                at Module._load (node:internal/modules/cjs/loader:1022:12)
                at Module.require (node:internal/modules/cjs/loader:1231:19)
                at require (node:internal/modules/helpers:179:18)
                at Object.<anonymous> (/home/user/odymaterialy.skauting.cz/node_modules/glob-stream/index.js:9:16)
                at Module._compile (node:internal/modules/cjs/loader:1369:14)
                at Module._extensions..js (node:internal/modules/cjs/loader:1427:10),
            pipeline: [Pipeline],
            drains: null,
            byteLength: [Function: defaultByteLength],
            map: null,
            afterWrite: [Function: bound afterWrite],
            afterUpdateNextTick: [Function: bound updateWriteNT]
          },
          _transformState: TransformState {
            data: null,
            afterTransform: [Function: bound afterTransform],
            afterFinal: null
          },
          _transform: [Function: writeFile],
          [Symbol(shapeMode)]: false,
          [Symbol(kCapture)]: false
        },
        _isPipeline: true,
        _pipelineMissing: 2,
        _writeCallback: null,
        _finalCallback: null,
        _ondata: [Function: bound _pushData],
        _onend: [Function: bound _pushEnd],
        _ondrain: [Function: bound _continueWrite],
        _onfinish: [Function: bound _maybeFinal],
        _onerror: [Function: bound destroy],
        _onclose: [Function: bound destroy],
        [Symbol(shapeMode)]: false,
        [Symbol(kCapture)]: false
      },
      afterRead: [Function: bound afterRead],
      afterUpdateNextTick: [Function: bound updateReadNT]
    },
    _writableState: WritableState {
      stream: [Circular *1],
      queue: FastFIFO {
        hwm: 16,
        head: FixedFIFO {
          buffer: [ <16 empty items> ],
          mask: 15,
          top: 0,
          btm: 0,
          next: null
        },
        tail: FixedFIFO {
          buffer: [ <16 empty items> ],
          mask: 15,
          top: 0,
          btm: 0,
          next: null
        },
        length: 0
      },
      highWaterMark: 0,
      buffered: 0,
      error: [Circular *2],
      pipeline: null,
      drains: null,
      byteLength: [Function: defaultByteLength],
      map: null,
      afterWrite: [Function: bound afterWrite],
      afterUpdateNextTick: [Function: bound updateWriteNT]
    },
    _predestroy: [Function: predestroy],
    _transformState: TransformState {
      data: null,
      afterTransform: [Function: bound afterTransform],
      afterFinal: [Function: bound afterFinal]
    },
    _flush: [Function: flush],
    _read: [Function (anonymous)],
    [Symbol(shapeMode)]: false,
    [Symbol(kCapture)]: false
  },
  domainThrown: false
}

[grncdr]

Seems like this has nothing to do with merge-stream if it works in some of your Gulp 5 projects and not others. Could you try creating a minimal reproduction?

P.S. - does the destination .well-known directory in your sample code exist and/or need to be created?

[marekdedic]

Minimal repro: https://gist.github.com/marekdedic/ba636b4fcbf233d65a63b8c1720b057d

Run npm ci && gulp main to see the issue.

In the original code, the dist/.well-known folder needs to be created.

[grncdr]

This is not an issue with merge stream.

[nsmithdev]

I am receiving the same error when updating to gulp 5.0 The fix for me was to move 1 of my 4 file copy out of merge and run it normally. It was specifically the first one not sure why it runs differently from the other 3 other than specifically what it's copying or how it works based on what it's copying. Hope this helps someone.

[ghnp5]

Having the same issue here, after upgrading to Gulp 5.

I have a number of gulp.src(...).pipe(...) in a mergeStream().

I was trying to do one by one, and then in one big gulp.src([....]), I was trying file by file, in that array.

It seems that no matter what, if there are 3 items in that src array, it's enough to start triggering this crash. Seems a bit random, but "consistently", as in as soon as I make it crash, it always crashes.

Definitely something related to mergeStream(), as otherwise, they never fail if outside a merged stream.

I might just have to separate all the streams into a big series of many functions, as mergeStream() no longer seems to work well with Gulp 5.

EDIT - Ah.. just found this - https://github.com/gulpjs/gulp/issues/2802

The merge-stream package is no longer supported, as it isn't a properly behaving stream. You can instead use the ordered-read-stream package that the gulpjs team maintains, which has the same interface/usage as merge-stream.

[marekdedic]

Yes, my solution was to switch to ordered-read-stream as well, recommend to anyone migrating to gulp 5.

[grncdr]

I'm glad to hear there's an easy solution.

I don't know for sure if anybody else depends on the {end: false} behavior and I'm not particularly inclined to find out by changing it.

[ghnp5]

Thanks.

Yeah, it worked for me, by just changing:

const mergeStream = require('merge-stream');

to:

const mergeStream = require('ordered-read-streams');

(and installing ordered-read-streams, of course)