Secrets not working on Kubernetes cluster

ivor-grisel commented 2 months ago

Describe the issue

Setup:

Github repo, which is used by >
ArgoCD to deploy Kestra
Everything runs on an AWS EKS cluster

I've used the following files:

values.yaml defined here: https://github.com/kestra-io/helm-charts/blob/master/charts/kestra/values.yaml. With the addition of the value extraSecretEnvFrom as described here: https://kestra.io/docs/how-to-guides/kubernetes-secrets (and everything is copy paste from these pages)
Dependency in the Chart.yaml is set to the latest version (0.18.0)
The flow as defined here: https://kestra.io/docs/how-to-guides/kubernetes-secrets (again, copy & paste)

The end result, when executing this flow, doesn't show any information in regards to password and username (seen screenshot next post) (edited)

Environment

Kestra Version: 0.18.0 (or at least, that's the Helm chart that I'm using)
Operating System (OS/Docker/Kubernetes): AWS, Kubernetes
Java Version (if you don't run kestra in Docker):

Additional info:

This was a clean install of Kestra. So literally copy & paste of the above files
Not sure if Docker in Docker is necessary, but that is running "as-is" in the values.yaml. If root in Docker was necessary, that was unclear from the documentation

Importance

We are currently looking at Kestra to replace our managed file transfer solution. We are not a paying customer yet. This is just something that I noticed. From my end, the priority of this is quite low, as we can test Kestra without it working.

Ben8t commented 2 months ago

@paulgrainger85 do you see anything wrong going on here 🤔 ?

paulgrainger85 commented 2 months ago

I can't see anything obviously wrong - is it possible to share the config of the service you have created, along with a mock-up of the secrets you've applied? I have followed the instructions above myself and the secret is picked up as expected

Ben8t commented 1 month ago

@ivor-grisel can you share more details on the configuration you have created as mentionned by Paul ?

ivor-grisel commented 1 month ago

@paulgrainger85 and @Ben8t so sorry for not getting back to you earlier, but I didn't receive any notification that you replied. I'll check to see if my settings for this incident are right.

And yes, I can share those. Here's the values.yaml that I've set up (so this is actually a complete clean install, using version 0.19):

image:
  image: kestra/kestra
  tag: latest
  pullPolicy: Always

imagePullSecrets: []
# - name: regcred

### Configuration for Kestra Deployment
## The following 4 sections can be used to set up the Kestra configuration
## Please note the secrets defined here are purposely for configuration your Kestra server
## To use secrets in your flows, please see below under "Application Secrets"

### This creates a config map of the Kestra configuration
configuration: {}
# Example: Setting the plugin defaults for the Docker runner
#   kestra:
#     plugins:
#       configurations:
#         - type:  io.kestra.plugin.scripts.runner.docker.Docker
#           values:
#             volume-enabled: true

### This will create a Kubernetes Secret for the values provided
## This will be appended to kestra-secret with the key application-secrets.yml
secrets: {}
# Example: Store your postgres backend credentials in a secret
#   secrets:
#     kestra:
#       datasources:
#         postgres:
#           username: pguser
#           password: mypass123
#           url: jdbc:postgresql://pghost:5432/db

### Load Kestra configuration from existing secret
## Here this assumes the secret is already deployed and the following apply:
## 1. The secret type is "Opaque"
## 2. The secret has a single key
## 3. The value of the secret is the base64 encoded Kestra configuration.
externalSecret: {}
  #secretName: secret-name
#key: application-kestra.yml

### configuration files
## This option allows you to reference existing local files to configure Kestra, e.g.
configurationPath:
# configurationPath: /app/application.yml,/app/application-secrets.yml

### Kestra executable
executable: /app/kestra

### Deployments
deployments:
  webserver:
    enabled: false
    kind: Deployment
    replicaCount: 1
    command: "server webserver"
    labels: {}
    annotations: {}
    resources: {}
    nodeSelector: {}
    tolerations: []
    affinity: {}
    strategy: {}
    podSecurityContext: {}
    securityContext: {}
    terminationGracePeriodSeconds: 30
    extraContainers: []
    extraEnv: []
    autoscaler:
      enabled: false
      minReplicas: 1
      maxReplicas: 3
      extra: {}
      metrics: []
      #  - type: Resource
      #    resource:
      #      name: cpu
      #      target:
      #        type: Utilization
      #        averageUtilization: 50
      #  - type: Resource
      #    resource:
      #      name: memory
      #      target:
      #        type: AverageValue
      #        averageValue: 100Mi

  executor:
    enabled: false
    kind: Deployment
    command: "server executor"

  indexer:
    enabled: false
    kind: Deployment
    command: "server indexer"

  scheduler:
    enabled: false
    kind: Deployment
    command: "server scheduler"

  worker:
    enabled: false
    kind: Deployment
    command: "server worker {{- if $.Values.deployments.worker.workerThreads }}--thread={{ .Values.deployments.worker.workerThreads }}{{- end }}"
    terminationGracePeriodSeconds: 60
    # By default, we start a number of threads of two times the number of available processors, use 'workerThreads' to configure a different value.
    #workerThreads: 128

  standalone:
    enabled: true
    kind: Deployment
    command: "server standalone {{- if $.Values.deployments.standalone.workerThreads }}--worker-thread={{ .Values.deployments.standalone.workerThreads }}{{- end }}"
    terminationGracePeriodSeconds: 60
    # By default, we start a number of threads of two times the number of available processors, use 'workerThreads' to configure a different value.
    #workerThreads: 128

# EE only - Define additional group of workers.
# Must be used in addition to default workers (in standalone or separate worker deployment).
workerGroup:
  workergroupname: # name of the worker group
    enabled: false
    kind: Deployment
    command: "server worker {{- if $.Values.workerGroup.workergroupname.workerThreads }}--thread={{ .Values.workerGroup.workergroupname.workerThreads }}{{- end }} --worker-group={{ .WorkerGroup }}"
    terminationGracePeriodSeconds: 60
    # By default, we start a number of threads of two times the number of available processors, use 'workerThreads' to configure a different value.
    #workerThreads: 128

# EE only - the Kestra Kubernetes Operator
operator:
  enabled: false
  image: registry.kestra.io/docker/kestra-operator
  apiKey: ""
  basicAuth: ""

# for io.kestra.core.tasks.scripts.Bash task or io.kestra.plugin.scripts.*, attach a docker dind container in order to isolate in a container
# every command launch
dind:
  enabled: true
  image:
    image: docker
    tag: dind-rootless
    pullPolicy: IfNotPresent
  socketPath: /dind/
  tmpPath: /tmp/
  resources: {}
  args:
    - --log-level=fatal
    - --group=1000
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
  extraVolumeMounts: []
  extraEnv: []

### Kafka
kafka:
  enabled: false
  listeners:
    client:
      protocol: PLAINTEXT
    controller:
      protocol: PLAINTEXT
    interbroker:
      protocol: PLAINTEXT
    external:
      protocol: PLAINTEXT

### ElasticSearch
elasticsearch:
  enabled: false
  clusterName: "es-kestra"
  createCert: false
  protocol: http
  esConfig:
    elasticsearch.yml: |
      xpack.security.enabled: false

### Minio
minio:
  enabled: true
  auth:
    rootUser: please-change-me
    rootPassword: its-not-a-secret
  defaultBuckets: kestra
  resources:
    requests:
      memory: 512Mi

### Postgresql
postgresql:
  enabled: true
  auth:
    database: kestra
    username: kestra
    password: kestra

  primary:
    persistence:
      enabled: true
      size: 8Gi

### Service
service:
  type: ClusterIP
  port: 8080
  loadBalancerIP: ""
  annotations: {}
  management:
    enabled: false
    port: 8081

### Ingress
ingress:
  enabled: false
  className: ""
  annotations: {}
  # kubernetes.io/ingress.class: nginx
  # kubernetes.io/tls-acme: "true"
  hosts:
    - host: chart-example.local
      paths:
        - path: /
          pathType: ImplementationSpecific
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

### Global Deployement
nameOverride: ""
serviceAccountName: ""

### Annotations for deployments
annotations: {}
podAnnotations: {}
initContainers: []

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
#   memory: 128Mi

terminationGracePeriodSeconds: 30

nodeSelector: {}

tolerations: []

affinity: {}

extraVolumeMounts: []

extraVolumes: []

extraEnv:
  - name: SECRET_DB_USERNAME
    valueFrom:
      secretKeyRef:
        name: db-creds
        key: username
  - name: SECRET_DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-creds
        key: password

extraContainers: []

# https://kestra.io/docs/administrator-guide/configuration/others#kestravariablesenv-vars-prefix
extraConfigMapEnvFrom:
# - name: my-existing-configmap-no-prefix
# - name: my-existing-configmap-with-prefix
#   prefix: KESTRA_

extraSecretEnvFrom:
# - name: my-existing-no-prefix
# - name: my-existing-with-prefix
#   prefix: SECRET_

### Application Secrets
## To pass in secrets of your applications, services etc to Kestra please see the following:
## https://kestra.io/docs/how-to-guides/kubernetes-secrets

podSecurityContext: {}
# fsGroup: 2000

securityContext:
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  runAsUser: 1000
  runAsGroup: 1000

### Startup / Readiness / Liveness probe config.
### ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/
readinessProbe:
  enabled: true
  path: /health
  port: management
  initialDelaySeconds: 0
  periodSeconds: 5
  timeoutSeconds: 3
  successThreshold: 1
  failureThreshold: 3
  httpGetEnabled: true
  httpGetExtra: {}

livenessProbe:
  enabled: true
  path: /health
  port: management
  initialDelaySeconds: 0
  periodSeconds: 5
  timeoutSeconds: 3
  successThreshold: 1
  failureThreshold: 3
  httpGetEnabled: true
  httpGetExtra: {}

startupProbe:
  enabled: true
  path: /health
  port: management
  initialDelaySeconds: 1
  periodSeconds: 1
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 120
  httpGetEnabled: true
  httpGetExtra: {}

# -- Extra Kubernetes Manifests
extraManifests: []```

Here is the secret (also taken from your documentation), which I've manually added to our Kubernetes environment (so not inside the folder which is in our Github for Kestra - you can see that because of the managedFields and such):
```apiVersion: v1
data:
  password: Y0dGemMzZHZjbVFLCg==
  username: WVdSdGFXNEsK
kind: Secret
metadata:
  creationTimestamp: '2024-10-07T07:03:06Z'
  managedFields:
    - apiVersion: v1
      fieldsType: FieldsV1
      fieldsV1:
        f:data:
          .: {}
          f:password: {}
          f:username: {}
        f:type: {}
      manager: dashboard-api
      operation: Update
      time: '2024-10-07T07:03:06Z'
  name: db-creds
  namespace: kestra
  resourceVersion: '48061088'
  uid: 9883c824-41c2-4239-851b-5ae6b1bd3d16
type: Opaque

And the flow:

namespace: company.team
tasks:
  - id: hello
    type: io.kestra.plugin.core.output.OutputValues
    values:
      username: "{{ secret('DB_USERNAME') }}"
      password: "{{ secret('DB_PASSWORD') }}"

When running the flow in Kestra, you can see that there is no outputvalues shown:

kestra-io-4960-outputvalues

kestra-io / kestra