'No response from server' when using writeStream

[Thetortuga]

Hi,

I have the following issue when I try to use the writeStream. I am not sure if this is a bug or me not knowing how to use stream.

[0] end for 2024-07-01T14:06:00.004Z
[0] end 2024-07-01T14:06:00.004Z
[0] end sftp 2024-07-01T14:06:00.004Z
[0] node:events:491
[0]       throw er; // Unhandled 'error' event
[0]       ^
[0] 
[0] Error: No response from server
[0]     at cleanupRequests (/home/nicolas/dev/yakapush/node_modules/ssh2/lib/protocol/SFTP.js:2730:15)
[0]     at SFTP.push (/home/nicolas/dev/yakapush/node_modules/ssh2/lib/protocol/SFTP.js:191:7)
[0]     at onCHANNEL_CLOSE (/home/nicolas/dev/yakapush/node_modules/ssh2/lib/utils.js:50:13)
[0]     at ChannelManager.cleanup (/home/nicolas/dev/yakapush/node_modules/ssh2/lib/utils.js:200:7)
[0]     at Socket.<anonymous> (/home/nicolas/dev/yakapush/node_modules/ssh2/lib/client.js:831:21)
[0]     at Socket.emit (node:events:513:28)
[0]     at TCP.<anonymous> (node:net:313:12)
[0] Emitted 'error' event on WriteStream instance at:
[0]     at emitErrorNT (node:internal/streams/destroy:151:8)
[0]     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
[0]     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

The code used to produce the error is following:

        try {
          const now = new Date()

          const writeStream = sftp.createWriteStream(now.toISOString())
          await writeStream.write('[')
          /* Writing stuff */
          await writeStream.write(',')

          await writeStream.end(']')
          console.log('end', now)
       /* WORKAROUND:  const timeout = () => new Promise(resolve => setTimeout(resolve, 500))
          await timeout() */

          console.log('end sftp', now)
        } catch (error) {
          console.error(error)
        } finally {
          await sftp.end()
        }

I apply the workaround found in this issue, which fix my problem: https://github.com/theophilusx/ssh2-sftp-client/issues/371

NodeJS: 18.9.0 Module: 10.0.3

[theophilusx]

I cannot tell what you are attempting to do based on that small bit of code snippet. However, I suspect your on completely the wrong track.

The sftp.createWriteStream() and sftp.createReadStream() are 'escape hatch' functions which are rarely used. In 99% of situations, you would use either sftp.put() or sftp.get(), both of which can accept stream objects directly as the source, for sftp.put or the destination for sftp.get.

You would need to send a minimal complete working code example for me to understand more clearly what your trying to do.

[Thetortuga]

Thanks for you answer. I use this because I want to upload a file based on a database request which is paginated. I retrieve a fixed number of result on each loop and I write them to the stream. Do you think there is a better solution ? Here is a more complete code to reproduce my issue:

try {
          const now = new Date()
          console.log('Start cron', now)

          await sftp.connect({
            host: 'localhost',
            port: '2222',
            username: 'demo',
            password: 'demo'
          })
          console.log('connected', now)
          const writeStream = sftp.createWriteStream(now.toISOString())
          await writeStream.write('[')

          let x = 0
          let cont = true
          console.log('begin for', now)

          while (cont) {
            const messages = await scheduleController.retrieveMessages(schedule.id, x++, now, PAGESIZE)
            if (messages.results.length < PAGESIZE) cont = false
            console.log(messages.results.length)
            for (message of messages) {
              await writeStream.write(JSON.stringify(message.sentData))
            }
          }
          console.log('end for', now)
          await writeStream.end(']')
          console.log('end', now)
          const timeout = () => new Promise(resolve => setTimeout(resolve, 500))
          // await timeout()

          console.log('end sftp', now)
        } catch (error) {
          console.error(error)
        } finally {
          await sftp.end()
        }

Thanks.

[theophilusx]

You don't need to use createWriteStream as the put() method accepts a stream as the data source (note that the whole read/write stream thing can be a little confusing. It really just designates which 'end' of a stream your attached to).

Here is what I would do. Start by creating a script which just generates a file of the data from your db stream using a normal fs.writeStream. Once you habe that working, you know the stream of data from your database is working correctly. Then create a script which uploads a file to your sftp serber using put(). Use a fs.readStream to get the data and pass that as the source argument for the put() method. This will verify your sftp process is all working. Then combine the two scripts so that the source stream for the put() method is the stream receiving the data from the db.

The reason I suggest using put() is because with put, you get full promise/async support. If you use createWriteStream, you are accessing the low lebel ssh2 library, which is event based and does not support promises. This means you would need to add and manage event listeners, something which is handled automatically if you use the higher lebel API probided by ssh-sftp-client.

Note also that in your use case, the lower level event based API of the ssh2 module might actually be better. If you find sftp.put() doesn't work or becomes too hard to make work, you may want to look at using ssh2 directly. You may also find the example directory of the ssh2-sftp-client repo and the example in the readme showing how to gzip/unzip files on the fly useful as they provide some examples of using streams. The validate directory of the ssh2-sftp-client repo also has some example scripts showing just the use of ssh2 itself. Check the ssh2 docs for examples of how to use the low level credateWriteStream and event listeners. It isn't hard, but it is a completely different procedssing model compared to a promise based API.

[Thetortuga]

Thanks a lot for your response ! I finally managed to make it work thanks to pipeline and promisify.