Issue downloading large files

[sherif2011]

Hello,

I am loading some US States data, and it's working fine with all 50 states except the state of California. Size of the compressed file is ~3GB. It keeps downloading, but suddenly stops after around 1GB. Any clue? I tried different chunk sizes and different concurrency counts but in vain.

Below is a snippet of my code.

var Client = require("ssh2-promise");

const downloadOptions = { // options to define how the download should be processed concurrency: 640, chunkSize: 32768, step: async () => { // this callback runs unpon downloading each chunck totChunck = totChunck + 32; if (totChunck % 1024 === 0) console.log("read = " + Math.floor(totChunck / 1024).toString() + "MB"); } };

ssh = new Client(L2Connection); sftp = await ssh.sftp();

await sftp .fastGet("/Folder/" + file.filename, file.filename, downloadOptions) .catch(err => reject(err)); // Extract State File await ssh.close();

btw l2connection is { "host": "sftp.xxxxx.com", "port": "22", "username": "my-user", "password": "my-password", "identity": "identityFilePath", "keepaliveInterval" :2000, "keepaliveCountMax" :1000, "reconnect": "true", "reconnectTries": "50", "reconnectDelay": "5000"

Thanks, Sherif

[sanketbajoria]

@sherif2011

Can you please attach the detailed logging. You can generate detailed logging, by passing the debug function in ssh2 config option. For example: l2connection as { "host": "sftp.xxxxx.com", "port": "22", "username": "my-user", "password": "my-password", "identity": "identityFilePath", "debug": function(str) {console.log(str)} }

[sherif2011]

Interestingly, adding that debug line made the file complete downloading successfully (finishing 3.2G instead of just 1GB). I was hoping if it gets stuck so I would send you the debug data for the last chunck download attempt. Could, adding the debug, have delayed a little bit the process, (like allowing a buffer to free...)? Should I try adding a short delay between chunck downloads?

[sherif2011]

So basically adding that debug part made it load all big files successfully. Does this tell anything? I have my application running on a docker container in a detached mode. Should I just leave debugger on, or do you recommend a neater way? Thanks!

[sanketbajoria]

@sherif2011

it's weird, by adding debug, big files downloaded successfully. If you face any error in future, can you please attach the debug log. I wanted to check, if maybe ssh connection is getting break, during file transfer.

[sherifmankarious]

I am still facing the same issue. No problem downloading most US States. But when downloading CA, it stops data after transferring around 1GB of data. Adding detailed logging "debug": function(str) {console.log(str)} returns enormous amount of data, do you suggest adding any kind of filter?