get and fastGet stops copying with an error after 2GB

sagacitysite commented 9 months ago

Hi together,

I have a problem copying large files (> 2GB).

Setup

Node: v20.10.0
ssh2-sftp-client: 9.1.0
OSMC (Debian GNU/Linux 11 bullseye) on Raspberry Pi
Server: Primitive ftpd for Android (https://github.com/wolpi/prim-ftpd)

Problem

When copying a file that is larger than 2GB, the client stops copying after exactly 2GB, throwing an error message (see below).

Getting the file with FileZilla from the same server works without issues.

I couldn't find any related config for ssh2-sftp-client, nor for ssh2. Did I miss something?

Error

fastGet()

Error: fastGet->sftp: EFBIG: file too large, write Remote: /remote/path/to/my/video.mp4 Local: /local/path/to/my/video.mp4
    at SftpClient.fmtError (/some_path/node_modules/ssh2-sftp-client/src/index.js:82:22)
    at SftpClient.fastGet (/some_path/node_modules/ssh2-sftp-client/src/index.js:603:18)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

get()

Error: get: EFBIG: file too large, write /path/to/my/video.mp4
    at SftpClient.fmtError (/some_path/node_modules/ssh2-sftp-client/src/index.js:82:22)
    at WriteStream.<anonymous> (/some_path/node_modules/ssh2-sftp-client/src/index.js:530:18)
    at Object.onceWrapper (node:events:629:26)
    at WriteStream.emit (node:events:526:35)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

theophilusx commented 9 months ago

No idea. Never seen that error message before and it isn't one generated by ssh2-sftp-client. I suspect this is something specific to the Raspberry Pi OS. At any rate, this is specifric to the ssh2 module which ssh2-sftp-client depends on, so you will need to log an issue with ssh2. Unfortunately, you will likely need to provide a reproducible test case which only uses ssh2 in order to get some traction.

I have regularly used both ssh2 and ssh2-sftp-client to download large (multiple GB) files on Linux wiht no issues. In fact, I wrote ssh2-sftp-client specifically for an applicaiton which needed to download many very large weather data files each day. This make me further think it is likely an issue specific to raspberry pi OS (or possibly node v20, which I've not used much - the ssh2 author has mentioned a few times that he frequently runs into issues due to node chaning internals in new versions). It may be worth seeing if yo9u can reproduce the issue with node v18 just in case it is a node v20 issue.

Note that fastGet/fastPut often have problems due to their high dependency on capabilities of the sftp server. I would definitely focus on just using get()/put() until you know it works before trying fastGet()/fastPut().

To see how you could implement a simple test case just using ssh2 and whithout ssh2-sftp-client, have a look in the validation directory of the ssh2-sftp-client repository. The big problem you are going to have is that to help diagnose this issue, it needs to be reproducible. This will be a problem if it is pseicific to raspberry pi OS. I would also be chekcing some raspberry pi OS forums as I doubt very much this is anything to do with either ssh2-sftp-client or ssh2. It is likley either an issue with the Pi or with the version of node that runs on the Pi.

sagacitysite commented 9 months ago

Thanks a lot for your detailed reply! It's good to know that I didn't simply overlook something and that I can focus on looking in the direction of Nodejs and/or the OS. I'll keep you updated when I find out something!

sagacitysite commented 9 months ago

Ok, I found it. The Node Version was okay. It was related to my OS. By the way, I made a small mistake, I'm not using Raspberry Pi OS, but OSMC (but the Debian base system is correct) on my Raspberry Pi.

I tested my script on my laptop (Ubuntu based system) and it worked without any issues. After googling a while, I found out that I have probably the following problem on my Raspberry Pi:

» https://en.wikipedia.org/wiki/2_GB_limit

It would make much sense, since the copy process stops after exactly 2,147,483,647 bits and my system is 32-bit.

With this, I probably need to find a different OS solution for my Raspberry Pi. I close this issue.

Thanks again for your reply, that actually helped to look into the right direction!

theophilusx / ssh2-sftp-client