search

kolban-google / sftp-gcs

An implementation of an SFTP to Google Cloud Storage bridge.
Apache License 2.0
86 stars 34 forks source link

Unhandled Promise: Something with SSH2 #20

Closed noahjustinsure closed 2 years ago

noahjustinsure commented 2 years ago

Hi,

I am trying to set up a simple connection and it's possible my service account doesn't have the right permissions but I am not sure that's the case as I've given it some pretty wide ranging permissions and the client was able to create empty files... So it definitely has a connection and some form of access.

Here is how I have run the command:

node sftp-gcs.js --bucket gs://xxxx-sftp --port=xxxx --service-account-key-file=/home/xxxx/xxxx.json --debug=debug --user=xxxx --password=xxxx

Here are the permissions I have provided (one section for each set of permissions in each role I have assigned to this service account):

customRole

logging.buckets.copyLogEntries
logging.buckets.create
logging.buckets.delete
logging.buckets.get
logging.buckets.list
logging.buckets.undelete
logging.buckets.update
logging.cmekSettings.get
logging.cmekSettings.update
logging.exclusions.create
orgpolicy.policy.get
resourcemanager.projects.get
storage.buckets.create
storage.buckets.delete
storage.buckets.get
storage.buckets.list
storage.multipartUploads.abort
storage.multipartUploads.create
storage.multipartUploads.listParts
storage.objects.create
storage.objects.delete
storage.objects.get
storage.objects.list
storage.objects.update

LoggingAdmin

logging.buckets.copyLogEntries
logging.buckets.create
logging.buckets.delete
logging.buckets.get
logging.buckets.list
logging.buckets.undelete
logging.buckets.update
logging.cmekSettings.get
logging.cmekSettings.update
logging.exclusions.create
logging.exclusions.delete
logging.exclusions.get
logging.exclusions.list
logging.exclusions.update
logging.fields.access
logging.locations.get
logging.locations.list
logging.logEntries.create
logging.logEntries.download
logging.logEntries.list
logging.logMetrics.create
logging.logMetrics.delete
logging.logMetrics.get
logging.logMetrics.list
logging.logMetrics.update
logging.logServiceIndexes.list
logging.logServices.list
logging.logs.delete
logging.logs.list
logging.notificationRules.create
logging.notificationRules.delete
logging.notificationRules.get
logging.notificationRules.list
logging.notificationRules.update
logging.operations.cancel
logging.operations.get
logging.operations.list
logging.privateLogEntries.list
logging.queries.create
logging.queries.delete
logging.queries.get
logging.queries.list
logging.queries.listShared
logging.queries.share
logging.queries.update
logging.queries.updateShared
logging.sinks.create
logging.sinks.delete
logging.sinks.get
logging.sinks.list
logging.sinks.update
logging.usage.get
logging.views.access
logging.views.create
logging.views.delete
logging.views.get
logging.views.list
logging.views.listLogs
logging.views.listResourceKeys
logging.views.listResourceValues
logging.views.update
resourcemanager.projects.get
resourcemanager.projects.list

LogWriter

logging.logEntries.create

StorageAdmin

firebase.projects.get
orgpolicy.policy.get
resourcemanager.projects.get
resourcemanager.projects.list
storage.buckets.create
storage.buckets.createTagBinding
storage.buckets.delete
storage.buckets.deleteTagBinding
storage.buckets.get
storage.buckets.getIamPolicy
storage.buckets.list
storage.buckets.listTagBindings
storage.buckets.setIamPolicy
storage.buckets.update
storage.multipartUploads.abort
storage.multipartUploads.create
storage.multipartUploads.list
storage.multipartUploads.listParts
storage.objects.create
storage.objects.delete
storage.objects.get
storage.objects.getIamPolicy
storage.objects.list
storage.objects.setIamPolicy
storage.objects.update

Here are some errors I am encountering: List Doesn't work:

sftp> ls
(node:16740) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'EOF' of undefined                                                                                           
    at SFTP.<anonymous> (/home/noahgary/sftp-gcs/sftp-gcs.js:663:69)                          
    at SFTP.emit (events.js:314:20)                                                           
    at 12 (/home/noahgary/sftp-gcs/node_modules/ssh2/lib/protocol/SFTP.js:3115:15)            
    at SFTP.push (/home/noahgary/sftp-gcs/node_modules/ssh2/lib/protocol/SFTP.js:278:11)      
    at CHANNEL_DATA (/home/noahgary/sftp-gcs/node_modules/ssh2/lib/server.js:747:23)          
    at 94 (/home/noahgary/sftp-gcs/node_modules/ssh2/lib/protocol/handlers.misc.js:859:16)    
    at Protocol.onPayload (/home/noahgary/sftp-gcs/node_modules/ssh2/lib/protocol/Protocol.js:2024:10)                                                                                      
    at ChaChaPolyDecipherNative.decrypt (/home/noahgary/sftp-gcs/node_modules/ssh2/lib/protocol/crypto.js:753:26)                                                                           
    at Protocol.parsePacket [as _parse] (/home/noahgary/sftp-gcs/node_modules/ssh2/lib/protocol/Protocol.js:1993:25)                                                                        
    at Protocol.parse (/home/noahgary/sftp-gcs/node_modules/ssh2/lib/protocol/Protocol.js:292:16)                                                                                           
(node:16740) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)                                          
(node:16740) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

put command works for empty file: but throws error:

sftp> put emptytest.file
Uploading emptytest.file to /emptytest.file
emptytest.file                                                                                                                                             100%    0     0.0KB/s   00:00
2021-12-16T20:15:37.378Z debug: [sftp-gcs] Client connected!
2021-12-16T20:15:37.436Z debug: [sftp-gcs] authentication: method=none
2021-12-16T20:15:37.436Z debug: [sftp-gcs] We have at least a user to match
2021-12-16T20:15:40.870Z debug: [sftp-gcs] Client disconnected
2021-12-16T20:15:53.185Z debug: [sftp-gcs] Client connected!
2021-12-16T20:15:53.193Z debug: [sftp-gcs] authentication: method=none
2021-12-16T20:15:53.193Z debug: [sftp-gcs] We have at least a user to match
2021-12-16T20:15:56.672Z debug: [sftp-gcs] authentication: method=password
2021-12-16T20:15:56.673Z debug: [sftp-gcs] Client authenticated!
2021-12-16T20:15:56.676Z debug: [sftp-gcs] Client SFTP session
2021-12-16T20:15:56.678Z debug: [sftp-gcs] REALPATH<1>: path: "."
2021-12-16T20:15:56.678Z debug: [sftp-gcs] Returning "/"
2021-12-16T20:16:02.617Z debug: [sftp-gcs] OPEN<2>: filename: "/emptytest.file", flags: w
2021-12-16T20:16:02.617Z debug: [sftp-gcs] Converted "/emptytest.file" to "emptytest.file"
2021-12-16T20:16:02.617Z debug: [sftp-gcs] Opening file for WRITE
2021-12-16T20:16:02.621Z debug: [sftp-gcs] CLOSE<3>: handle: 0
2021-12-16T20:16:02.621Z debug: [sftp-gcs] Closing GCS write stream
2021-12-16T20:16:02.926Z debug: [sftp-gcs] Detected an error with writeStream to the GCS file: TypeError: Cannot read property 'OK' of undefined

put command for non-empty file doesn't work:

sftp> put another.file
Uploading another.file to /another.file
another.file                                                                                                                                                 0%    0     0.0KB/s   --:-- ETAConnection to localhost closed by remote host.
Connection closed
2021-12-16T20:19:09.735Z debug: [sftp-gcs] Client connected!
2021-12-16T20:19:09.748Z debug: [sftp-gcs] authentication: method=none
2021-12-16T20:19:09.748Z debug: [sftp-gcs] We have at least a user to match
2021-12-16T20:19:12.919Z debug: [sftp-gcs] authentication: method=password
2021-12-16T20:19:12.919Z debug: [sftp-gcs] Client authenticated!
2021-12-16T20:19:12.964Z debug: [sftp-gcs] Client SFTP session
2021-12-16T20:19:12.967Z debug: [sftp-gcs] REALPATH<1>: path: "."
2021-12-16T20:19:12.967Z debug: [sftp-gcs] Returning "/"
2021-12-16T20:19:17.813Z debug: [sftp-gcs] OPEN<2>: filename: "/another.file", flags: w
2021-12-16T20:19:17.814Z debug: [sftp-gcs] Converted "/another.file" to "another.file"
2021-12-16T20:19:17.814Z debug: [sftp-gcs] Opening file for WRITE
2021-12-16T20:19:17.819Z debug: [sftp-gcs] WRITE<3>: handle: 0, offset 0: data.length=10
/home/noahgary/sftp-gcs/sftp-gcs.js:534
                        sftpStream.status(reqId, STATUS_CODE.OK);
                                                             ^

TypeError: Cannot read property 'OK' of undefined
    at /home/noahgary/sftp-gcs/sftp-gcs.js:534:62
    at afterWrite (/home/noahgary/sftp-gcs/node_modules/readable-stream/lib/_stream_writable.js:469:3)
    at processTicksAndRejections (internal/process/task_queues.js:86:21)

Please forgive me for any IAM issues as I'm pretty new to GCP and still learning how to manage all of this...

noahjustinsure commented 2 years ago

I used @karatekaneen 's typescript fork and was able to get this running in no time. Same permissions and everything. https://github.com/karatekaneen/sftp-gcs

kolban-google commented 2 years ago

@noahjustinsure OOOh ... thank you for the post. It appears a few days ago, Github came into my project and force upgraded some of the package.json files. Specifically, it changed a dependency from a 0.x version to a 1.x version. This introduced a breaking change. Researching now.

noahjustinsure commented 2 years ago

@kolban-google Oh my! Silly Github. Thanks so much for your work! I noticed the syntax for ssh2 is different now:

const {
  Server,
  sftp: {
    OPEN_MODE,
    STATUS_CODE,
  },
} = require('ssh2');

rather than what we have now:

const ssh2 = require('ssh2');

const STATUS_CODE = ssh2.SFTP_STATUS_CODE;
kolban-google commented 2 years ago

@noahjustinsure Ahh ... TY sir am dedicated to fixing this now ... here is the Pull Request that broke the show

https://github.com/kolban-google/sftp-gcs/pull/19

The fault is ALWAYS and ultimately mine. Apparently Github made the recommendation to upgrade and another committer (this week) accepted. He issued a pull request ... and it was ME who didn't test it. That means that the fault is mine (bad moderator).

noahjustinsure commented 2 years ago

No worries at all @kolban-google you've done great work.