gulp exits with 'Error in plugin' when gulp-sftp used with gulp-concat and gulp.watch

[u01jmg3]

gulp.task('compile:js:remote', function(){
    return gulp.src(srcJs, {base: src})
        .pipe(plumber({
            errorHandler: onError
        }))
        .pipe(removelogs())
        .pipe(uglify({outSourceMap: false}))
        .pipe(order([
            '**/**/jquery.js',
            '**/**/jquery.ui.js',
            '**/**/custom.js',
            '**/**/modernizr.js',
            '**/**/jquery.fancybox.js',
            '**/**/*.js'
        ]))
        .pipe(concat('all.min.js'))
        .pipe(gulp.dest(distScripts))
        .pipe(sftp({
            host: host,
            auth: auth,
            remotePath: remotePath + '/' + distScripts
        }))        
    ;
});

compile:js:remote is triggered after a change (gulp.watch) is made to source JavaScript. It should re-minify my code and upload the resulting file. However, gulp-sftp is erroring out with the following. Only just introduced gulp-concat to this task but previously gulp-sftp was working without issue when I wasn't combining my JavaScript files. I've tried {buffer: false} but on the first run where files for this project (43 files) are uploaded gulp-sftp has no problem dealing with all.min.js (which is only 459KB).

[14:17:33] Starting 'compile:js:remote'...
[14:17:37] Authenticating with private key.
[14:17:37] SFTP :: SFTP session closed

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error in plugin '←[36mgulp-sftp←[39m'
SFTP abrupt closure
    at SFTP.<anonymous> (C:\Users\com265\AppData\Roaming\npm\node_modules\gulp-s
ftp\index.js:148:44)
    at SFTP.emit (events.js:92:17)
    at ChannelStream.<anonymous> (C:\Users\com265\AppData\Roaming\npm\node_modul
es\gulp-sftp\node_modules\ssh2\lib\SFTP\SFTPv3.js:58:10)
    at ChannelStream.g (events.js:180:16)
    at ChannelStream.emit [as _emit] (events.js:117:20)
    at ChannelStream.emit (C:\Users\com265\AppData\Roaming\npm\node_modules\gulp
-sftp\node_modules\ssh2\lib\Channel.js:638:12)
    at Parser.<anonymous> (C:\Users\com265\AppData\Roaming\npm\node_modules\gulp
-sftp\node_modules\ssh2\lib\Channel.js:60:20)
    at Parser.g (events.js:180:16)
    at Parser.emit (events.js:92:17)
    at Parser.parsePacket (C:\Users\com265\AppData\Roaming\npm\node_modules\gulp
-sftp\node_modules\ssh2\lib\Parser.js:644:12)

[gtg092x]

The easiest solution is to switch to streams. The closure is an issue with ssh2 - please check out the known issues section of the readme.

[u01jmg3]

I have already read your readme and tried using buffer: false but that just caused more errors.

I don't quite understand what the problem is and what I should be changing my task to do? I know roughly it has something to do with buffers and streams.

Should I try using one of the plugins Lars Kappert suggests such as gulp-streamify?

[gtg092x]

Streamify should take care of it. I would encourage you to bring the issue up with the author of ssh2 and cite the sudden disconnect when having large buffers.

[u01jmg3]

https://github.com/gtg092x/gulp-sftp/blob/master/package.json#L36

@gtg092x: @mscdex has suggested updating this project to the latest version of ssh2 (v0.3.3) but that you may face some issues with backwards compatibility. His comment is here: https://github.com/mscdex/ssh2/issues/153#issuecomment-47988427

Do you think you could make this update? Let me know if you want me to contribute a pull request and look into this

[gtg092x]

Sure thing - I'll make the change tomorrow evening - thank you!

[gtg092x]

@mscdex @u01jmg3 - I've updated my ssh2 and my tests are still getting abrupt connection closings. I've got a branch ready with the changes in place, but it looks like that issue still exists and can be recreated using large buffers via large transfers. I haven't dug into it, but my guess is that it's a memory management error via pagenet.

I could be wrong though. If this issue is on my end, I'll gladly accept pull requests that can upload large files without relying on streamBuffers. I just can't find anywhere that I'm forcing a sftp disconnect.

In the mean time, refer to gulp-streamify if we can't solve your issue.

[gtg092x]

Refer to https://github.com/gtg092x/gulp-sftp/tree/sshbump for expected changes

[mscdex]

Do you have an actual minimal reproducible test somewhere?

[gtg092x]

Yes - though you might have different memory limitations because you're not on the same machine. I always get an error uploading the attached file. If you need more details - auth information, etc. Let me know.

[mscdex]

I think I see what the problem is. OpenSSH limits the size of its SFTP packets to 256kb. So sending the whole contents at one time (~3mb) surpasses this and OpenSSH rejects the data, despite the SFTP spec allowing for a full, unsigned 32-bit packet length. I'm assuming that means OpenSSH always buffers entire SFTP packets, no matter what the actual request is (which is really silly).

I'm not sure what the other implementations do, but I guess I could detect OpenSSH and enforce the same limit in that case.

[gtg092x]

@mscdex - I appreciate it; I'm dodging some work by acting like it's not my problem, but your module does far more work than mine and supports way more use cases. Thanks for stepping to make that change. Let me know when that's in and I'll update my branch with the latest version.

[mscdex]

@gtg092x @u01jmg3 This should be fixed now as of mscdex/ssh2@2603c4dc8. Please give the master branch a try.

[u01jmg3]

@gtg092x: will you be updating gulp-sftp to use the latest version of ssh2?

[mscdex]

@u01jmg3 I haven't published a new version that includes the previously mentioned change yet, I will publish a new version once others have also tested it to make sure it works for them as well.

[u01jmg3]

Okay, no problem

[mscdex]

You should be able to test it by changing to the gulp-sftp module directory and then doing npm install https://github.com/mscdex/ssh2/tarball/master. Then re-run your script that you were previously having issues with.

[u01jmg3]

I re-worked my script to get around this issue so will have to revert back for testing - will let you know how it goes. Thanks for the tip

[gtg092x]

@mscdex The version on your master branch works without disconnecting me. Thank you for your help! Let me know when you've bumped your version with npm and I'll do the same on this end.

[u01jmg3]

All good for me too - thanks again to you both @mscdex @gtg092x

I'll await the version bump

[mscdex]

I've now published ssh2 v0.3.4. Thanks all.

[gtg092x]

Updated on this end and on npm - closing