Unable to find/access a system host key, using the application default host key

[Aldigor]

Hi,

Please……. I have problem with sftp-gcs, when I run it in k8s. I've got the key for the service account in the json file which was downloaded when creating the key. The service account has StorageAdmin role on the bucket and roles/logging-admin

>kubectl logs sftp-gsc-6dd46fb85c-7vpp5
2022-02-23T12:03:34.399Z warn: [sftp-gcs] Unable to find/access a system host key, using the application default host key.
2022-02-23T12:03:34.412Z info: [sftp-gcs] ****************************************
2022-02-23T12:03:34.412Z info: [sftp-gcs] *** Google Cloud Storage SFTP Server ***
2022-02-23T12:03:34.412Z info: [sftp-gcs] ****************************************
2022-02-23T12:03:34.412Z info: [sftp-gcs] Using bucket: gs://sftp-cs-dev
2022-02-23T12:03:34.412Z info: [sftp-gcs] Listening on port 9022
2022-02-23T12:03:34.412Z info: [sftp-gcs] Username: test
2022-02-23T12:03:34.413Z info: [sftp-gcs] Password: ********
2022-02-23T12:03:34.413Z info: [sftp-gcs] Public key file: Not set
2022-02-23T12:03:34.413Z info: [sftp-gcs] Service account key file: gcloud-key.json
(node:1) UnhandledPromiseRejectionWarning: Error: 7 PERMISSION_DENIED: The caller does not have permission
    at Object.callErrorFromStatus ([/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:31:26](mailto:/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:31:26))
    at Object.onReceiveStatus ([/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:180:52](mailto:/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:180:52))
    at Object.onReceiveStatus ([/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:365:141](mailto:/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:365:141))
    at Object.onReceiveStatus ([/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181](mailto:/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181))
    at [/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:182:78](mailto:/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:182:78)
    at processTicksAndRejections (internal/process/task_queues.js:79:11)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Docker file is:

FROM node:12
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .

EXPOSE 9022
CMD ["node", "sftp-gcs.js", "--bucket", "sftp-cs-dev", "--user", "test", "--password", "password", "--service-account-key-file", "gcloud-key.json", "--port", "9022"]

Yaml for k8s is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sftp-gsc
  namespace: default
  labels:
    app: sftp-gsc
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: sftp-gsc
  template:
    metadata:
      labels:
        app: sftp-gsc
    spec:
      containers:
      - name: sftp-gsc
        image: eu.gcr.io/**********/sftp-gsc:latest
        ports:
        - containerPort: 9022
          name: http
          protocol: TCP
        resources:
          limits:
            memory: 256Gi
          requests:
            memory: 64Mi

Thank you for help

Robert

[kolban-google]

Howdy my friend. At first blush, this looks like a problem with the SFTP-GCS server application (this package) not being able to read the gcloud-key.json file. I'm not a K8S hero ... so I'm going to need your help in debugging in that area. Do I understand that this file is "local" in your file system and "burnt" / "copied" into the Docker container? My first set of questions for us to examine are:

1. What are the Linux file permissions on that file?
2. What Linux userid is the demon running under within the container within K8S?

For example, if the file on the file system is defined to have permissions

-rw------- root root

That would say that only root has permissions to read the file. However, if the demon runs as a userid other than root, it would not be able to open the file and we would get the error described. This does feel like a mystery though as I have always felt that Container hosted apps run as root and that should have no permission checks. We should try an experiment by setting your gcloud-key.json to 777

chmod 777 gcloud-key.json

and re-building the container and trying again. I'm not suggesting this as the ultimate solution, but it will verify that it is as Linux file permissions issue.

If it would help, I'll make myself available for a screen share at a convenient time for both of us ... kolban@google.com

Neil