theophilusx
commented
2 years ago You are running an old version of ssh2-sftp-client, please upgrade to 9.0.1 and see if you can reproduce your issue.
Closed dan-emmanuel closed 2 years ago
theophilusx
commented
2 years ago You are running an old version of ssh2-sftp-client, please upgrade to 9.0.1 and see if you can reproduce your issue.
dan-emmanuel
commented
2 years ago On it i'll update u
theophilusx
commented
2 years ago Note I have just pushed out version 9.0.2
dan-emmanuel
commented
2 years ago hello i've updated the package to 9.0.2 and I still have the problem of sending empty files. but : the error fmtError is no more triggered
dan-emmanuel
commented
2 years ago hello @theophilusx any updates on that ticket ?
theophilusx
commented
2 years ago I cannot reproduce your issue. You need to supply a full minimal working script which reproduces your error in order for me to diagnose any further. Also, you nee to let me know what version of node you are using and what platform you are running on.
dan-emmanuel
commented
2 years ago unfortunately, i can't repeat the problem on purpose...... here's a copy of my pacakgeJson : ` "dependencies": { "@azure/storage-blob": "^12.3.0", "@elastic/elasticsearch": "^7.10.0", "@line/bot-sdk": "^7.4.0", "@nestjs/common": "^8.2.3", "@nestjs/core": "^8.2.3", "@nestjs/platform-express": "^8.2.3", "@nestjs/schedule": "^1.0.2", "@nestjs/swagger": "^5.2.0", "@nestjs/typeorm": "^8.0.2", "@types/ip": "^1.1.0", "@types/ssh2-sftp-client": "^5.3.2", "@types/uuid": "^8.3.4", "aes256": "^1.1.0", "atob": "^2.1.2", "cache-manager": "^3.6.1", "class-validator": "^0.13.2", "cli-color": "^2.0.0", "connect": "^3.7.0", "csvtojson": "^2.0.10", "dateformat": "^4.5.1", "dot-object": "^2.1.4", "dotenv": "^8.2.0", "fs": "0.0.1-security", "html-minifier": "^4.0.0", "http": "0.0.1-security", "httpntlm": "^1.7.6", "https": "^1.0.0", "js-md4": "^0.3.2", "json-2-csv": "^3.17.0", "json-csv": "^3.0.6", "json2csv": "^5.0.6", "jsonwebtoken": "^8.5.1", "lodash.groupby": "^4.6.0", "moment": "^2.29.1", "moment-timezone": "^0.5.34", "mysql": "^2.18.1", "mysqldump": "^3.2.0", "nestjs-i18n": "^8.0.10", "node-fetch": "^2.6.1", "node-html-parser": "^5.3.3", "object-path": "^0.11.8", "openpgp": "^4.10.9", "performance-now": "^2.1.0", "reflect-metadata": "^0.1.13", "rimraf": "^3.0.2", "rxjs": "^7.4.0", "soap": "^0.35.0", "ssh2-sftp-client": "^7.0.1", "swagger-ui-express": "^4.3.0", "typeorm": "^0.2.29", "util": "^0.12.3", "uuid": "^8.3.1", "winston": "^3.3.3", "xml2js": "^0.4.23", "xml2json": "^0.12.0", "xmlbuilder": "^15.1.1", "zlib": "^1.0.5" }, "devDependencies": { "@nestjs/cli": "^8.2.4", "@nestjs/schematics": "^7.2.5", "@nestjs/testing": "^8.2.3", "@types/cache-manager": "^3.4.3", "@types/cli-color": "^2.0.0", "@types/dateformat": "^3.0.1", "@types/express": "^4.17.13", "@types/html-minifier": "^4.0.2", "@types/html-pdf": "^2.2.0", "@types/jest": "26.0.10", "@types/json2csv": "^5.0.3", "@types/jsonwebtoken": "^8.5.8", "@types/lodash": "^4.14.178", "@types/node": "^14.14.10", "@types/object-path": "^0.11.1", "@types/supertest": "^2.0.8", "@types/xml2js": "^0.4.9", "@typescript-eslint/eslint-plugin": "3.9.1", "@typescript-eslint/parser": "3.9.1", "eslint": "^7.32.0", "eslint-config-prettier": "^6.15.0", "eslint-plugin-import": "^2.20.1", "ip": "^1.1.5", "jest": "26.4.2", "prettier": "^1.19.1", "supertest": "^4.0.2", "ts-jest": "26.2.0", "ts-loader": "^6.2.1", "ts-node": "^9.0.0", "tsconfig-paths": "^3.9.0", "typescript": "^4.5.4" }, "jest": { "moduleFileExtensions": [ "js", "json", "ts" ], "rootDir": "src", "testRegex": ".spec.ts$", "transform": { "^.+\.(t|j)s$": "ts-jest" }, "coverageDirectory": "../coverage", "testEnvironment": "node" }
`
the code is running on an azure webapp :
here's the snippet that generate the error :
const sftpConfig = config.sftp; const sftp = new SFTPClient(); await sftp.connect({ host: sftpConfig.host, port: sftpConfig.port, username: sftpConfig.username, password: sftpConfig.password, }); await sftp.put(Buffer.from(data, 'utf-8'), fileName);
data is type : string, so is fileName
in attachment you'll find one the file at 0k in the sftp and the same file at 224 b in the blob azure
theophilusx
commented
2 years ago if you are unable to provide a script which can reproduce the issue, I cannot help. I have to be able to reproduce the issue in order to diagnose it. I don't even know what version of node your using.
Just to be clear, as I don't have a clear picture of your architecture, you are running ssh2-sftp-client inside node and not a web browser aren't you? (just a little concerned when you say it is running inside a web app on azure. The moduel only works when run inside the node environment as it depends on node specific APIs which are not found in browser JS environments).
At any rate, if you are unable to provide a script which reproduces the issue, I cannot help and you will need to diagnose this problem yourself.
dan-emmanuel
commented
2 years ago i've send you the package json of the app. yess of course the module is executed in a nest.js environment (not a web browser) the web azure is the name of the service that contenarize the app. the version of node is 14 the code
theophilusx
commented
2 years ago I sent a response to this some days ago, but it does not seem to have appeared in this thread.
Thanks for confirming you are running in a node environment. While that may have seemed like a silly question, you would be surprised with the number of issues logged where people were trying to use this module inside a browser environment, hence why I had to verify.
Without a script to reproduce the issue, I cannot help diagnose the problem. If you are unable to supply a script which reproduces the issue you are seeing, you will have to diagnose the problem yourself.
The issue you are seeing is almost certainly something specific to your code and environment. This module is downloaded over 200k times per week and you are the only one reporting this issue. I have script which take data from a buffer and transfer it up to a remote sftp server with no problems. I cannot reproduce the issue you are seeing.
The first thing I would do is add a debug function (you can do this via the connect options object). That might provide more insight into what is going on.
The next thing I would do is write a very simple script which only uses ssh2-sftp-client (no next.js etc) and see if you can get that to upload to an sftp server. The trick will be to make it as minimal as possible. If that also fails, I would then write the same script, but only using ssh2 (which ssh2-sftp-client uses to perform the low level sftp stuff). If that also fails, then you might find it useful checking with that project. If the simple ssh2-sftp-client script works, then you know the issue is some compatibility issue between ssh2-sftp-client (or ssh2 more likely) and one of the other libraries your using. I would start slowly adding back the other libraries/modules you are using, testing after each addition, until you can identify where the problem is.
Once you have more insight into the problem, working out where to go will be much easier. Right now, your main objective is to find out was is the minimum script you need to reproduce the error.
dan-emmanuel
commented
2 years ago hello, we finally found the origin of our problem take a chair and a beer, and be ready to laugh.......
we have uploaded our code on 2 instances of the same server.... the code was using a nest built-in cron system
now it appears that several times a day, the 2 instances run the same code at the same second, so they both tried to write in the same azure blob a .csv file; since they tried to write at the same second, then the second one overwrote the content of the first instance..... and at the end of the day..... the file had a 0 ko file.......
so we fixed the problem by putting the job in an HTTP root, and we programmed a server-less function that is running at is running 1 per hour, and the load balancer decides which of the instance execute the job.
since not several instances of the same code are no more executed at the same time..... no more problems...... 2 months to find the root causes, but it's finally done........
for the next ones that are coming lately here...... be sure that you're code is only executed once at the same m.second 😅
theophilusx
commented
2 years ago The move to cloud based containers and architecture which runs multiple instances in order to improve performance can be challenging to manage. We tend to think about problems from a synchronous rather than asychronous perspective. I recall in the mid 1990s when everyone was adding page view counters to their web pages, nearly all of them were 'broken' because those writing them didn't consider that more than one client could be viewing a web page at the same time.
Modern architectures can be complex and difficult to debug. I actually find it important when designing systems to include debugging and maintenance requirements in the design in addition to the core functionality. We must design systems to be maintained, which means incorporating logging, diagnostics and debugging features into the design and not ad hoc additional later as problems are discovered.
Glad you were able to make progress. The smallest problems can sometimes be the most frustrating and can easily take the shine of an otherwise interesting project.
theophilusx
commented
2 years ago dan-emmanuel @.***> writes:
i've send you the package json of the app.
Unfortunately, that is of little help to me. I cannot reproduce the issue you are seeing based on that information.
yess of course the module is executed in a nest.js environment (not a web browser) the web azure is the name of the service that contenarize the app. the version of node is 14 the code
You might say 'yes of course', but I have had many issues people have raised where it turned out the problem was because they were trying to run the code in JS running in a browser! I too thought it was obvious you could only use the module in a node environment, but it seems many others don't get that.
The issue you are seeing is almost certainly something specific to your code and environment. This module is downloaded over 200k times per week and you are the only one reporting this issue. I have script which take data from a buffer and transfer it up to a remote sftp server with no problems. I cannot reproduce the issue you are seeing.
It is quite likely the issue is due to how azure has implemented their node environment or something obscure triggered by one of the numberous different modules you are loading. There have been other 'issues' found with node modules running in both azure and amazon environments due to differences these environments have. To track down what this might be, you need to start with the simplest and most basic script you can get to work - only loading the minimum needed modules i.e. start wiht just ssh2-sftp-client and its direct dependencies. Add the additional functionality bit by bit, only adding additional modules as needed. Build back up until you trigger the issue again and you should have a better clue as to where the problem is arising.
It is also possible this could be some issue introduced by nest.js. I would also try a simple ssh2-sftp-script which just runs inside node on azure wihtout nest.js and see if that works to upload to the sftp server. For all we know, this could be an issue caused by nest.js itself.
I would start by adding a debug function and looking at the output to see if that gives you any clue (you can pass in a debug function as one of the config options passed to connect). I would then write a very simple basic version of your script which is able to just take a buffer and write it to a remote sftp server outside azure. Once you know that is working, then transfer it to azure and run it there. The key will be to make it as minimal and simple as possible. If it works, then gradually start adding functionality until you trigger whatever it is that is preventing the upload from working. If it doesn't work, I would try using both streams instead of buffers to see if that works (streams tend to be more efficient than buffers anyway and it all gets converted to streams at some point regardless, so streams are less work).
If it still doesn't work, then I would try writing a very simple script just using ssh2. You will have to write it using an event model rather than promises/async. This will put you at the lowest level and may help identify what the cause of your issue is.
I would also check on the azure related forums. It is highly likely this is something specific to azure's node environment and a good chance others have encountered something similar.
If azure gives you the choice, I would also try and target node 16 or node 18. There were some issues with node 14, which were resolved in latter versions of v14. In particular, there were some chagnes in the network stack in early v14 which were later rolled back (sorry, cannot recall the specific patch levels). It was one of the reasons the author of the ssh2 module (which ssh2-sftp-client depends on) re-wrote the whole module (ssh2 1.x is a complete rewrite from the previous 0.8.x version). I realise, being azure, you may not have much choice on what node environment you can use, I would recommend at least v16 if at all possible.
The key will be starting with something as basic and simple as possible with as few other packages as you can get away with. Then start adding bits slowly until you find what is triggering the issue. You will then have something to focus on to find why it fails.
As you cannot provide a script and I cannot reproduce the issue, I don't think there is much else I can do. You will have to diagnose this issue yourself.
Goo luck.
theophilusx
commented
2 years ago Unless you can provide a simple script which reproduces the issue your encountering, I cannot do much. I need to be able to reproduce your issue and so far, all of my attempts to do so have failed.
Odds are, this is an issue specific to your environment. I would try your script against different sftp servers and from different platforms.
dan-emmanuel @.***> writes:
hello i've updated the package to 9.0.2 and I still have the problem of sending empty files. but : 1- the error fmtError is no more triggered 2- now some files are not even sent
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.
public static async sendToSftp(data: string, fileName: string) { try { const sftpConfig = config.sftp; const sftp = new SFTPClient(); await sftp.connect({ host: sftpConfig.host, port: sftpConfig.port, username: sftpConfig.username, password: sftpConfig.password, }); await sftp.put(Buffer.from(data, 'utf-8'), fileName) ... } catch (err) { this.logger.error(
Error in saving the csv file '${fileName}' in SFTP server); this.logger.error(err); } }at fmtError (at fmtError (/node_modules/ssh2-sftp-client/src/utils.js:55:18): /node_modules/ssh2-sftp-client/src/utils.js:55) at WriteStream. (at WriteStream. (/node_modules/ssh2-sftp-client/src/index.js:728:18): /node_modules/ssh2-sftp-client/src/index.js:728)
at Object.onceWrapper (at Object.onceWrapper (events.js:421:26): events.js:421)
at WriteStream.emit (at WriteStream.emit (events.js:314:20): events.js:314)
at errorOrDestroy (at errorOrDestroy (internal/streams/destroy.js:108:12): internal/streams/destroy.js:108)
at onwriteError (at onwriteError (_stream_writable.js:424:5): _stream_writable.js:424)
at onwrite (at onwrite (_stream_writable.js:445:5): _stream_writable.js:445)
at (at /node_modules/ssh2/lib/protocol/SFTP.js:3699:14: /node_modules/ssh2/lib/protocol/SFTP.js:3699)
at Object.cb (at Object.cb (/node_modules/ssh2/lib/protocol/SFTP.js:478:13): /node_modules/ssh2/lib/protocol/SFTP.js:478)
at 101 (at 101 (/node_modules/ssh2/lib/protocol/SFTP.js:2621:11): /node_modules/ssh2/lib/protocol/SFTP.js:2621)
at SFTP.push (at SFTP.push (/node_modules/ssh2/lib/protocol/SFTP.js:278:11): /node_modules/ssh2/lib/protocol/SFTP.js:278)
at CHANNEL_DATA (at CHANNEL_DATA (/node_modules/ssh2/lib/client.js:525:23): /node_modules/ssh2/lib/client.js:525)
at 94 (at 94 (/node_modules/ssh2/lib/protocol/handlers.misc.js:859:16): /node_modules/ssh2/lib/protocol/handlers.misc.js:859)
at Protocol.onPayload (at Protocol.onPayload (/node_modules/ssh2/lib/protocol/Protocol.js:2025:10): /node_modules/ssh2/lib/protocol/Protocol.js:2025)
at GenericDecipherNative.decrypt (at GenericDecipherNative.decrypt (/node_modules/ssh2/lib/protocol/crypto.js:1269:26): /node_modules/ssh2/lib/protocol/crypto.js:1269)
at Protocol.parsePacket [as _parse] (at Protocol.parsePacket [as _parse] (/node_modules/ssh2/lib/protocol/Protocol.js:1994:25): /node_modules/ssh2/lib/protocol/Protocol.js:1994)
package.json : "@types/ssh2-sftp-client": "^5.3.2", package-lock.json : "node_modules/ssh2-sftp-client": { "version": "7.2.2", "resolved": "https://registry.npmjs.org/ssh2-sftp-client/-/ssh2-sftp-client-7.2.2.tgz", "integrity": "sha512-qZYivU1zezyRomCf+TtsCYVAsc0TDQWzxJMMUN8NknEPonm2TYGxJAzrW8acUh2ILYgA0ZPOJElLV/qp9nRVYQ==", "dependencies": { "concat-stream": "^2.0.0", "promise-retry": "^2.0.1", "ssh2": "^1.6.0" }, "engines": { "node": ">=10.24.1" } },