search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.88k stars 9.16k forks source link

[bitnami/airflow] Mount my own ssh-dir(/.ssh) after 18.0.6? #27561

Closed mujiannan closed 1 month ago

mujiannan commented 3 months ago

Name and Version

bitnami/airflow 18.0.6+

What is the problem this feature will solve?

In the past versions: 

git:
  clone:
    extraVolumeMounts:
    - name: git-ssh-key
      mountPath: "/.ssh"

Key commit: 78249480860eefd30da45311f0dfcfc341429bfe Ineffective in 18.0.6+:
image

How can I mount my own ssh-dir(/.ssh) after 18.0.6?
Actually, I found a workaround for 18.0.6+: 

git:
  clone:
    extraEnvVars:
    - name: GIT_SSH_COMMAND
      value: "ssh -i /opt/bitnami/git/.ssh/id_rsa -o StrictHostKeyChecking=no"
    extraVolumeMounts:
    - name: git-ssh-key
      mountPath: "/opt/bitnami/git/.ssh"
  sync:
    extraEnvVars:
    - name: GIT_SSH_COMMAND
      value: "ssh -i /opt/bitnami/git/.ssh/id_rsa -o StrictHostKeyChecking=no"
    extraVolumeMounts:
    - name: git-ssh-key
      mountPath: "/opt/bitnami/git/.ssh"

Can we design a better solution?

7358

What is the feature you are proposing to solve the problem?

Airflow - Git clone & sync with user-defined ssh-secret.

What alternatives have you considered?

No response

carrodher commented 3 months ago

Thank you for bringing this issue to our attention. We appreciate your involvement! If you're interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

github-actions[bot] commented 2 months ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

kitos9112 commented 2 months ago

Just encountered this issue while trying to mount the .ssh folder in my initContainers. I was skimming through https://github.com/bitnami/charts/pull/25219

I assume that's to allow manual edits over the GUI for Airflow? By doing so you broke the previous mount of SSH keys from K8s secrets :(

Nickmman commented 2 months ago

I'm also encountering this issue, I've tried mixing and matching the volumeMounts with subPaths and setting GIT_SSH_COMMAND with verbosity but it seems to have no effect. No output is generated from the clone-repositories container other than:

 02:29:57.63 INFO  ==>
 02:29:57.72 INFO  ==> Welcome to the Bitnami git container
 02:29:57.72 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
 02:29:57.72 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
 02:29:57.72 INFO  ==>
Stream closed EOF for dev/airflow-web-5fbbd6757d-gbkl4 (clone-repositories)
kitos9112 commented 2 months ago

I'm also encountering this issue, I've tried mixing and matching the volumeMounts with subPaths and setting GIT_SSH_COMMAND with verbosity but it seems to have no effect. No output is generated from the clone-repositories container other than:

 02:29:57.63 INFO  ==>
 02:29:57.72 INFO  ==> Welcome to the Bitnami git container
 02:29:57.72 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
 02:29:57.72 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
 02:29:57.72 INFO  ==>
Stream closed EOF for dev/airflow-web-5fbbd6757d-gbkl4 (clone-repositories)

What does your helm values config look like? I did manage to work around this issue by using the extraEnvVar directive.

Nickmman commented 2 months ago

@kitos9112 This is what my values look like:

auth:
  existingSecret: airflow-auth-bitnami
  username: admin
diagnosticMode:
  enabled: false
git:
  clone:
    extraEnvVars:
    - name: GIT_SSH_COMMAND
      value: ssh -vvvi /opt/bitnami/git/.ssh/id_rsa -o StrictHostKeyChecking=no
    extraVolumeMounts:
    - mountPath: /opt/bitnami/git/.ssh/id_rsa
      name: git-ssh-key
      subPath: id_rsa
  dags:
    enabled: true
    repositories:
    - branch: dev
      name: git
      repository: redacted
  sync:
    extraEnvVars:
    - name: GIT_SSH_COMMAND
      value: ssh -vvvi /opt/bitnami/git/.ssh/id_rsa -o StrictHostKeyChecking=no
    extraVolumeMounts:
    - mountPath: /opt/bitnami/git/.ssh/id_rsa
      name: git-ssh-key
      subPath: id_rsa
global:
  storageClass: ceph-block
ingress:
  annotations:
    cert-manager.io/cluster-issuer: dns-issuer
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
    nginx.ingress.kubernetes.io/ssl-passthrough: "false"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  enabled: true
  hostname: redacted
  ingressClassName: nginx-internal
  tls: true
postgresql:
  auth:
    existingSecret: airflow-auth-bitnami
redis:
  auth:
    existingSecret: airflow-auth-bitnami
scheduler:
  extraVolumes:
  - name: git-ssh-key
    secret:
      defaultMode: 256
      secretName: svc-iac-ssh-key
  resources:
    limits:
      cpu: "1.5"
      memory: 768Mi
    requests:
      cpu: "1"
      memory: 512Mi
web:
  extraVolumes:
  - name: git-ssh-key
    secret:
      defaultMode: 256
      secretName: svc-iac-ssh-key
worker:
  extraVolumes:
  - name: git-ssh-key
    secret:
      defaultMode: 256
      secretName: svc-iac-ssh-key
Nickmman commented 2 months ago

@kitos9112 Can you share the extraEnvVar value for getting around this issue so I can test in my environment?

kitos9112 commented 2 months ago

@kitos9112 Can you share the extraEnvVar value for getting around this issue so I can test in my environment?

Hey!

Yes, here they go. At the root of the helm chart, I mount the same volumes across all airflow pods

extraVolumes:
  - name: git-ssh-key
    secret:
      secretName: git-ssh-key-secret
      defaultMode: 256
extraVolumeMounts:
  - name: git-ssh-key
    mountPath: /.ssh

Then, I simply inject those couple of env vars. For example for the clone init container

git:
  clone:
    command: []
    args: []
    extraVolumeMounts:
      - name: git-ssh-key
        mountPath: "/opt/bitnami/git/.ssh"
    extraEnvVars:
      - name: GIT_SSH_COMMAND
        value: "ssh -i /opt/bitnami/git/.ssh/id_rsa -o StrictHostKeyChecking=no"

I think you shouldn't set the Git SSH key mountpath with the filename, but just the directory as I do.

EDIT: I posted my comment a bit early.

Nickmman commented 2 months ago

@kitos9112 Unfortunately I still have the same issue. What version of the chart are you using?

kitos9112 commented 2 months ago

@Nickmman I think I'm using 18.3.8 - Are you sure the SSH key is available in your K8s cluster as a secret? Does it get mounted in other containers? What logs do you get in your worker or scheduler nodes?

Nickmman commented 2 months ago

@Nickmman I think I'm using 18.3.8 - Are you sure the SSH key is available in your K8s cluster as a secret? Does it get mounted in other containers? What logs do you get in your worker or scheduler nodes?

Now that I updated my version to 18.3.8, I get verbose logging in the containers. The SSH key is present and does get mounted. Now I have another error to figure out, regarding the SSH key itself. I'm getting back Load key "/opt/bitnami/.ssh/id_rsa": error in libcrypto, I don't suppose you encountered this issue as well?

kitos9112 commented 2 months ago

Contributor

Nope. Is your key correctly stored and coded in the secret? What happens if you retrieve the secret content using kubectl, decode from base64 and try to open it with ssh-keygen/ssh-agentor similar tools?

Nickmman commented 2 months ago

Contributor

Nope. Is your key correctly stored and coded in the secret? What happens if you retrieve the secret content using kubectl, decode from base64 and try to open it with ssh-keygen/ssh-agentor similar tools?

Looks like I had a malformed SSH key, I got it working now, thanks for your help! In the end, these are the helm values that made everything work:

git:
  clone:
    extraEnvVars:
    - name: GIT_SSH_COMMAND
      value: ssh -i /opt/bitnami/.ssh/id_rsa -o StrictHostKeyChecking=no
    extraVolumeMounts:
    - mountPath: /opt/bitnami/.ssh
      name: git-ssh-key
  dags:
    enabled: true
    repositories:
    - branch: dev
      name: git
      repository: git@redacted
  sync:
    extraEnvVars:
    - name: GIT_SSH_COMMAND
      value: ssh -i /opt/bitnami/.ssh/id_rsa -o StrictHostKeyChecking=no
    extraVolumeMounts:
    - mountPath: /opt/bitnami/.ssh
      name: git-ssh-key
scheduler:
  extraVolumes:
  - name: git-ssh-key
    secret:
      defaultMode: 256
      secretName: svc-iac-ssh-key
web:
  extraVolumes:
  - name: git-ssh-key
    secret:
      defaultMode: 256
      secretName: svc-iac-ssh-key
worker:
  extraVolumes:
  - name: git-ssh-key
    secret:
      defaultMode: 256
      secretName: svc-iac-ssh-key
github-actions[bot] commented 1 month ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 1 month ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.