Corrupt json file

[janajri]

Attempting to download a JSON file from a Cerberus FTP server (version 5). Upon inspection, it appears that a quote is being dropped before the second key. I was able to pull the file with no issues on my linux install using sftp command and validate it as valid JSON.

While I understand this might be related to the server's protocol it is in fact outside of my control. Any help would be greatly appreciated

"stat":0.0905846153846,top_ftrxavg_wkc13":0.37284615384'

Attempting to JSON.parse the raw downloaded string results in the following error:

SyntaxError: Unexpected token t in JSON at position 65535

[mscdex]

How are you downloading the file?

[janajri]

  host: CLIENT_SERVER,
  user: CLIENT_USERNAME,
  username: CLIENT_USERNAME,
  password: CLIENT_PASSWORD,
  algorithms: {
    compress: ["none"],
     cipher: [
       'aes128-ctr',
       'aes192-ctr',
       'aes256-ctr',
       'aes128-gcm',
       'aes128-gcm@openssh.com',
       'aes256-gcm',
       'aes256-gcm@openssh.com',
       'aes256-cbc'
     ]
   }
}

const file = await sftp.get(rootPath + '/' + next.name, null, 'ascii')
file.pipe(fs.createWriteStream('./test.json'))```

Sorry if I didn't clarify, but I'm using one of your other libraries where this is a dependency.

[mscdex]

Have you tried using ssh2 directly? It seems you are using a third party module that perhaps builds on top of ssh2, since ssh2 does not export any Promises.

[janajri]

My bad, yes -- I am using this: https://github.com/jyu213/ssh2-sftp-client/blob/master/src/index.js

But the issue doesn't seem to exist here as I was able to validate the appropriate inputs.

I will retry with ssh2 directly and report back -- it's been a week since I last attempted.

[janajri]

var conn = new Client();
conn.on('ready', function() {
  console.log('Client :: ready');
  conn.sftp(function(err, sftp) {
    if (err) throw err;
    sftp.readdir('/backups', function(err, list) {
      if (err) throw err;
      const [next] = list.filter(s => s.filename.endsWith('json'))
      const s = sftp.createReadStream('backups/' + next.filename, { encoding: 'ascii' })
      s.pipe(fs.createWriteStream('./test.json'))
      // conn.end();
    });
  });
}).connect(config);

f = require('./test.json') SyntaxError: /test.json: Unexpected token t in JSON at position 65535

I can confirm this is still an issue.

[mscdex]

Out of curiosity, does it work if you use sftp.fastGet() instead?

[janajri]

Woah -- that did work! Any thoughts on where the issue might lie while streaming the file?

[mscdex]

Not offhand, but that might help narrow down the issue if sftp.fastGet() works every time. Would it be possible to obtain a JSON (or any other kind) file that you're having problems with, so that I can see if I can duplicate it on my end? These kinds of issues are typically pretty difficult to duplicate...

[vitamindck]

I have the problem as reported above but it's not just JSON files. I'm using the vscode-sftp extension in Visual Studio Code and during download/diff of large UTF8 text files (200kb+) every 64101st character is removed. I have not experienced this corruption with files smaller than 64100 characters (UTF8 encoding).

I'm using a Mac/OSX 10.13.6 with node v8.11.3, npm 5.10.0, ssh2 ^0.6.1 (0.6.1 currently)

Please let me know if you need anymore information.

[mscdex]

@vitamindck Try replacing createReadStream() usage in vscode-sftp with fastGet() and see if the problem persists.

[vitamindck]

https://github.com/mscdex/ssh2-streams/issues/43 this is the same issue I'm having, and coincidentally(? or not..) I'm also using Cerberus FTP

[jnsbal]

I can confirm the issue still persists. With fastGet it works for me everytime