search

hashicorp / vault-helm

Helm chart to install Vault and other associated components.
Mozilla Public License 2.0
1.08k stars 873 forks source link

Vault agent container is not getting added to client application pod #770

Open anushanagireddy0430 opened 2 years ago

anushanagireddy0430 commented 2 years ago

After successful deployment of vault and consul to same namespace, nginx application is deployed to test vault agent injectior. Though correct annotations are added, We can see ngnix pod as 1/1. Vault-agent addon in nginx pod is not visible. Steps to reproduce the behavior:

  1. Install and vault and consul using a single helmchart
  2. Unseal and initialize vault
  3. Deploy nginx application using service DNS/FQDN, required role and policies in the same namespace

Logs/deployment yamls Injector logs image api-server logs

image

kubectl get deploy eops-vault-eccs-vault-helmcharts-agent-injector -n eops-dependencies -o yaml apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "1" meta.helm.sh/release-name: eops-vault meta.helm.sh/release-namespace: eops-dependencies creationTimestamp: "2022-07-29T17:18:50Z" generation: 1 labels: app.kubernetes.io/instance: eops-vault app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: eccs-vault-helmcharts-agent-injector component: webhook name: eops-vault-eccs-vault-helmcharts-agent-injector namespace: eops-dependencies resourceVersion: "7394620" uid: 5843e26c-7140-4299-b4e3-f7c85ccded51 spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/instance: eops-vault app.kubernetes.io/name: eccs-vault-helmcharts-agent-injector component: webhook strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: creationTimestamp: null labels: app.kubernetes.io/instance: eops-vault app.kubernetes.io/name: eccs-vault-helmcharts-agent-injector component: webhook spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution:

apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: "2022-08-03T11:40:29Z" generation: 1 name: app-example-deployment1 namespace: eops-dependencies resourceVersion: "7395004" uid: a5a34b00-c638-457e-8a7b-07a939fda7d8 spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: app-example1 strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: annotations: vault.hashicorp.com/agent-cache-enable: "true" vault.hashicorp.com/agent-cache-use-auto-auth-token: "true" vault.hashicorp.com/agent-configmap: my-configmap vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/agent-inject-command-db-creds: sh -c 'kill -HUP $(pidof client)' vault.hashicorp.com/agent-inject-status: update vault.hashicorp.com/agent-inject-token: "true" vault.hashicorp.com/log-level: debug vault.hashicorp.com/tls-secret: vault-server-tls creationTimestamp: null labels: app: app-example1 spec: containers:

Client application pod, nginx pod in this case should have 2 container nginx and vault-agent conatiner.

Environment

Chart values: Chart.yaml apiVersion: v2 appVersion: 1.9.0 dependencies:

Values.yaml

Available parameters and their default values for the Vault chart.

global:

enabled is the master enabled switch. Setting this to true or false

# will enable or disable all the components within this chart by default.
enabled: true
# Image pull secret to use for registry authentication.
# Alternatively, the value may be specified as an array of strings.
imagePullSecrets:
# imagePullSecrets:
  - name: docker-image
# TLS for end-to-end encrypted transport
tlsDisable: false
# If deploying to OpenShift
openshift: false
# enable monitoring to create dashboard in grafana
monitoring:
  enable: true
  label: "eccs-central-grafana-grafana"
# enable alerting to notify about vault status using prometheus alertmanager
alerting:
  enable: true
  label: "eccs-central-prometheus-prometheus"
# Create PodSecurityPolicy for pods
psp:
  enable: false
  # Annotation for PodSecurityPolicy.
  # This is a multi-line templated string map, and can also be set as YAML.
  annotations: |
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default,runtime/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName:  runtime/default

injector:

True if you want to enable vault agent injection.

enabled: true

replicas: 1

# Configures the port the injector should listen on
port: 8080

# If multiple replicas are specified, by default a leader will be determined
# so that only one injector attempts to create TLS certificates.
leaderElector:
  enabled: true

# If true, will enable a node exporter metrics endpoint at /metrics.
metrics:
  enabled: false

# External vault server address for the injector to use. Setting this will
# disable deployment of a vault server along with the injector.
externalVaultAddr: ""

# image sets the repo and tag of the vault-k8s image to use for the injector.
image:
  repository: "xxx.xx.x/eccs-vault-dockerimages"
  tag: "vault-k8s_0.14.1_0.18.0"
  pullPolicy: IfNotPresent

# agentImage sets the repo and tag of the Vault image to use for the Vault Agent
# containers.  This should be set to the official Vault image.  Vault 1.3.1+ is
# required.
agentImage:
  repository: "xxx.xx.x/eccs-vault-dockerimages"
  tag: "vault_1.9.0_0.18.0"

# The default values for the injected Vault Agent containers.
agentDefaults:
  # For more information on configuring resources, see the K8s documentation:
  # https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
  cpuLimit: "500m"
  cpuRequest: "250m"
  memLimit: "128Mi"
  memRequest: "64Mi"

  # Default template type for secrets when no custom template is specified.
  # Possible values include: "json" and "map".
  template: "map"

  # Default values within Agent's template_config stanza.
  templateConfig:
    exitOnRetryFailure: true
    staticSecretRenderInterval: ""

# Mount Path of the Vault Kubernetes Auth Method.
authPath: "auth/kubernetes"

# Configures the log verbosity of the injector.
# Supported log levels include: trace, debug, info, warn, error
logLevel: "info"

# Configures the log format of the injector. Supported log formats: "standard", "json".
logFormat: "standard"

# Configures all Vault Agent sidecars to revoke their token when shutting down
revokeOnShutdown: false

# namespaceSelector is the selector for restricting the webhook to only
# specific namespaces.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector
# for more details.
# Example:
# namespaceSelector:
#    matchLabels:
#      sidecar-injector: enabled
namespaceSelector: {}
# objectSelector is the selector for restricting the webhook to only
# specific labels.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-objectselector
# for more details.
# Example:
# objectSelector:
#    matchLabels:
#      vault-sidecar-injector: enabled
objectSelector: {}

# Configures failurePolicy of the webhook. The "unspecified" default behaviour deoends on the
# API Version of the WebHook.
# To block pod creation while webhook is unavailable, set the policy to `Fail` below.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#failure-policy
#
failurePolicy: Ignore

# Extra annotations to attach to the webhook
webhookAnnotations: {}

certs:
  # secretName is the name of the secret that has the TLS certificate and
  # private key to serve the injector webhook. If this is null, then the
  # injector will default to its automatic management mode that will assign
  # a service account to the injector to generate its own certificates.
  secretName: vault-server-tls

  # caBundle is a base64-encoded PEM-encoded certificate bundle for the CA
  # that signed the TLS certificate that the webhook serves. This must be set
  # if secretName is non-null, unless an external service like cert-manager is
  # keeping the caBundle updated.
  caBundle: ""

  # certName and keyName are the names of the files within the secret for
  # the TLS cert and private key, respectively. These have reasonable
  # defaults but can be customized if necessary.
  certName: vault.crt
  keyName: vault.key

resources:
  requests:
    memory: 256Mi
    cpu: 250m
  limits:
    memory: 256Mi
    cpu: 250m

# extraEnvironmentVars is a list of extra environment variables to set in the
# injector deployment.
extraEnvironmentVars: {}
  # KUBERNETES_SERVICE_HOST: kubernetes.default.svc

# Affinity Settings for injector pods
# This can either be multi-line string or YAML matching the PodSpec's affinity field.
# Commenting out or setting as empty the affinity variable, will allow
# deployment of multiple replicas to single node services such as Minikube.
affinity: |
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: {{ template "vault.name" . }}-agent-injector
            app.kubernetes.io/instance: "{{ .Release.Name }}"
            component: webhook
        topologyKey: kubernetes.io/hostname

# Toleration Settings for injector pods
# This should be either a multi-line string or YAML matching the Toleration array
# in a PodSpec.
tolerations: []

# nodeSelector labels for server pod assignment, formatted as a multi-line string or YAML map.
# ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
# Example:
# nodeSelector:
#   beta.kubernetes.io/arch: amd64
nodeSelector: {}

# Priority class for injector pods
priorityClassName: ""

# Extra annotations to attach to the injector pods
# This can either be YAML or a YAML-formatted multi-line templated string map
# of the annotations to apply to the injector pods
annotations: {}

# Extra labels to attach to the agent-injector
# This should be a YAML map of the labels to apply to the injector
extraLabels: {}

# Should the injector pods run on the host network (useful when using
# an alternate CNI in EKS)
hostNetwork: false

# Injector service specific config
service:
  # Extra annotations to attach to the injector service
  annotations: {}

# A disruption budget limits the number of pods of a replicated application
# that are down simultaneously from voluntary disruptions
podDisruptionBudget: {}
# podDisruptionBudget:
#   maxUnavailable: 1

server:

If not set to true, Vault server will not be installed. See vault.mode in _helpers.tpl for implementation details

enabled: true

# [Enterprise Only] This value refers to a Kubernetes secret that you have
# created that contains your enterprise license. If you are not using an
# enterprise image or if you plan to introduce the license key via another
# route, then leave secretName blank ("") or set it to null.
# Requires Vault Enterprise 1.8 or later.
enterpriseLicense:
  # The name of the Kubernetes secret that holds the enterprise license. The
  # secret must be in the same namespace that Vault is installed into.
  secretName: ""
  # The key within the Kubernetes secret that holds the enterprise license.
  secretKey: "license"

# Resource requests, limits, etc. for the server cluster placement. This
# should map directly to the value of the resources field for a PodSpec.
# By default no direct resource request is made.

image:
  repository: "xxx.xx.x/eccs-vault-dockerimages"
  tag: "vault_1.9.0_0.18.0"
  # Overrides the default Image Pull Policy
  pullPolicy: IfNotPresent

# Configure the Update Strategy Type for the StatefulSet
# See https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
updateStrategyType: "OnDelete"

# Configure the logging verbosity for the Vault server.
# Supported log levels include: trace, debug, info, warn, error
logLevel: ""

# Configure the logging format for the Vault server.
# Supported log formats include: standard, json
logFormat: ""

resources:
  requests:
    memory: 256Mi
    cpu: 250m
  limits:
    memory: 256Mi
    cpu: 250m

# Ingress allows ingress services to be created to allow external access
# from Kubernetes to access Vault pods.
# If deployment is on OpenShift, the following block is ignored.
# In order to expose the service, use the route section below
ingress:
  enabled: false
  labels: {}
    # traffic: external
  annotations: {}
    # |
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
    #   or
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"

  # Optionally use ingressClassName instead of deprecated annotation.
  # See: https://kubernetes.io/docs/concepts/services-networking/ingress/#deprecated-annotation
  ingressClassName: ""

  # As of Kubernetes 1.19, all Ingress Paths must have a pathType configured. The default value below should be sufficient in most cases.
  # See: https://kubernetes.io/docs/concepts/services-networking/ingress/#path-types for other possible values.
  pathType: Prefix

  # When HA mode is enabled and K8s service registration is being used,
  # configure the ingress to point to the Vault active service.
  activeService: true
  hosts:
    - host: chart-example.local
      paths: []
  ## Extra paths to prepend to the host configuration. This is useful when working with annotation based services.
  extraPaths: []
  # - path: /*
  #   backend:
  #     service:
  #       name: ssl-redirect
  #       port:
  #         number: use-annotation
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

# OpenShift only - create a route to expose the service
# The created route will be of type passthrough
route:
  enabled: false

  # When HA mode is enabled and K8s service registration is being used,
  # configure the route to point to the Vault active service.
  activeService: true

  labels: {}
  annotations: {}
  host: chart-example.local

# authDelegator enables a cluster role binding to be attached to the service
# account.  This cluster role binding can be used to setup Kubernetes auth
# method.  https://www.vaultproject.io/docs/auth/kubernetes.html
authDelegator:
  enabled: true

# extraInitContainers is a list of init containers. Specified as a YAML list.
# This is useful if you need to run a script to provision TLS certificates or
# write out configuration files in a dynamic way.
extraInitContainers: null
  # # This example installs a plugin pulled from github into the /usr/local/libexec/vault/oauthapp folder,
  # # which is defined in the volumes value.
  # - name: oauthapp
  #   image: "alpine"
  #   command: [sh, -c]
  #   args:
  #     - cd /tmp &&
  #       wget https://github.com/puppetlabs/vault-plugin-secrets-oauthapp/releases/download/v1.2.0/vault-plugin-secrets-oauthapp-v1.2.0-linux-amd64.tar.xz -O oauthapp.xz &&
  #       tar -xf oauthapp.xz &&
  #       mv vault-plugin-secrets-oauthapp-v1.2.0-linux-amd64 /usr/local/libexec/vault/oauthapp &&
  #       chmod +x /usr/local/libexec/vault/oauthapp
  #   volumeMounts:
  #     - name: plugins
  #       mountPath: /usr/local/libexec/vault

# extraContainers is a list of sidecar containers. Specified as a YAML list.
extraContainers: null

# shareProcessNamespace enables process namespace sharing between Vault and the extraContainers
# This is useful if Vault must be signaled, e.g. to send a SIGHUP for log rotation
shareProcessNamespace: false

# extraArgs is a string containing additional Vault server arguments.
extraArgs: ""

# Used to define custom readinessProbe settings
readinessProbe:
  enabled: true
  # If you need to use a http path instead of the default exec
  # path: /v1/sys/health?standbyok=true

  # When a probe fails, Kubernetes will try failureThreshold times before giving up
  failureThreshold: 2
  # Number of seconds after the container has started before probe initiates
  initialDelaySeconds: 5
  # How often (in seconds) to perform the probe
  periodSeconds: 5
  # Minimum consecutive successes for the probe to be considered successful after having failed
  successThreshold: 1
  # Number of seconds after which the probe times out.
  timeoutSeconds: 3
# Used to enable a livenessProbe for the pods
livenessProbe:
  enabled: true
  path: "/v1/sys/health?standbyok=true"
  # When a probe fails, Kubernetes will try failureThreshold times before giving up
  failureThreshold: 2
  # Number of seconds after the container has started before probe initiates
  initialDelaySeconds: 60
  # How often (in seconds) to perform the probe
  periodSeconds: 5
  # Minimum consecutive successes for the probe to be considered successful after having failed
  successThreshold: 1
  # Number of seconds after which the probe times out.
  timeoutSeconds: 3

# Optional duration in seconds the pod needs to terminate gracefully.
# See: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
terminationGracePeriodSeconds: 10

# Used to set the sleep time during the preStop step
preStopSleepSeconds: 5

# Used to define commands to run after the pod is ready.
# This can be used to automate processes such as initialization
# or boostrapping auth methods.
postStart: []
# - /bin/sh
# - -c
# - /vault/userconfig/myscript/run.sh

# extraEnvironmentVars is a list of extra environment variables to set with the stateful set. These could be
# used to include variables required for auto-unseal.
extraEnvironmentVars:
  VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
  # GOOGLE_REGION: global
  # GOOGLE_PROJECT: myproject
  # GOOGLE_APPLICATION_CREDENTIALS: /vault/userconfig/myproject/myproject-creds.json

# extraSecretEnvironmentVars is a list of extra environment variables to set with the stateful set.
# These variables take value from existing Secret objects.
extraSecretEnvironmentVars: []
  # - envName: AWS_SECRET_ACCESS_KEY
  #   secretName: vault
  #   secretKey: AWS_SECRET_ACCESS_KEY

# Deprecated: please use 'volumes' instead.
# extraVolumes is a list of extra volumes to mount. These will be exposed
# to Vault in the path `/vault/userconfig/<name>/`. The value below is
# an array of objects, examples are shown below.
extraVolumes:
  - type: secret
    name: vault-server-tls
  # - type: secret (or "configMap")
  #   name: my-secret
  #   path: null # default is `/vault/userconfig`

# volumes is a list of volumes made available to all containers. These are rendered
# via toYaml rather than pre-processed like the extraVolumes value.
# The purpose is to make it easy to share volumes between containers.
volumes: null
#   - name: plugins
#     emptyDir: {}

# volumeMounts is a list of volumeMounts for the main server container. These are rendered
# via toYaml rather than pre-processed like the extraVolumes value.
# The purpose is to make it easy to share volumes between containers.
volumeMounts: null
#   - mountPath: /usr/local/libexec/vault
#     name: plugins
#     readOnly: true

# Affinity Settings
# Commenting out or setting as empty the affinity variable, will allow
# deployment to single node services such as Minikube
# This should be either a multi-line string or YAML matching the PodSpec's affinity field.
affinity: |
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: {{ template "vault.name" . }}
            app.kubernetes.io/instance: "{{ .Release.Name }}"
            component: server
        topologyKey: kubernetes.io/hostname

# Toleration Settings for server pods
# This should be either a multi-line string or YAML matching the Toleration array
# in a PodSpec.
tolerations: []

# nodeSelector labels for server pod assignment, formatted as a multi-line string or YAML map.
# ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
# Example:
# nodeSelector:
#   beta.kubernetes.io/arch: amd64
nodeSelector: {}

# Enables network policy for server pods
networkPolicy:
  enabled: false
  egress: []
  # egress:
  # - to:
  #   - ipBlock:
  #       cidr: 10.0.0.0/24
  #   ports:
  #   - protocol: TCP
  #     port: 443

# Priority class for server pods
priorityClassName: ""

# Extra labels to attach to the server pods
# This should be a YAML map of the labels to apply to the server pods
extraLabels: {}

# Extra annotations to attach to the server pods
# This can either be YAML or a YAML-formatted multi-line templated string map
# of the annotations to apply to the server pods
annotations: {}

# Enables a headless service to be used by the Vault Statefulset
service:
  enabled: true
  # clusterIP controls whether a Cluster IP address is attached to the
  # Vault service within Kubernetes.  By default the Vault service will
  # be given a Cluster IP address, set to None to disable.  When disabled
  # Kubernetes will create a "headless" service.  Headless services can be
  # used to communicate with pods directly through DNS instead of a round robin
  # load balancer.
  # clusterIP: None

  # Configures the service type for the main Vault service.  Can be ClusterIP
  # or NodePort.
  #type: ClusterIP

  # The externalTrafficPolicy can be set to either Cluster or Local
  # and is only valid for LoadBalancer and NodePort service types.
  # The default value is Cluster.
  # ref: https://kubernetes.io/docs/concepts/services-networking/service/#external-traffic-policy
  externalTrafficPolicy: Cluster

  # If type is set to "NodePort", a specific nodePort value can be configured,
  # will be random if left blank.
  #nodePort: 30000

  # Port on which Vault server is listening
  port: 8200
  # Target port to which the service should be mapped to
  targetPort: 8200
  # Extra annotations for the service definition. This can either be YAML or a
  # YAML-formatted multi-line templated string map of the annotations to apply
  # to the service.
  annotations: {}

  #loadBalancerIP: {}
  #port: 30075
  #type: LoadBalancer

# This configures the Vault Statefulset to create a PVC for data
# storage when using the file or raft backend storage engines.
# See https://www.vaultproject.io/docs/configuration/storage/index.html to know more
dataStorage:
  enabled: true
  # Size of the PVC created
  size: 10Gi
  # Location where the PVC will be mounted.
  mountPath: "/vault/data"
  # Name of the storage class to use.  If null it will use the
  # configured default Storage Class.
  storageClass: eccs-vault-sc
  # Access Mode of the storage device being used for the PVC
  accessMode: ReadWriteOnce
  # Annotations to apply to the PVC
  annotations: {}

# This configures the Vault Statefulset to create a PVC for audit
# logs.  Once Vault is deployed, initialized and unseal, Vault must
# be configured to use this for audit logs.  This will be mounted to
# /vault/audit
# See https://www.vaultproject.io/docs/audit/index.html to know more
auditStorage:
  enabled: false
  # Size of the PVC created
  size: 10Gi
  # Location where the PVC will be mounted.
  mountPath: "/vault/audit"
  # Name of the storage class to use.  If null it will use the
  # configured default Storage Class.
  storageClass: eccs-vault-sc
  # Access Mode of the storage device being used for the PVC
  accessMode: ReadWriteOnce
  # Annotations to apply to the PVC
  annotations: {}

# Run Vault in "dev" mode. This requires no further setup, no state management,
# and no initialization. This is useful for experimenting with Vault without
# needing to unseal, store keys, et. al. All data is lost on restart - do not
# use dev mode for anything other than experimenting.
# See https://www.vaultproject.io/docs/concepts/dev-server.html to know more
dev:
  enabled: false

  # Set VAULT_DEV_ROOT_TOKEN_ID value
  devRootToken: "root"

# Run Vault in "standalone" mode. This is the default mode that will deploy if
# no arguments are given to helm. This requires a PVC for data storage to use
# the "file" backend.  This mode is not highly available and should not be scaled
# past a single replica.
standalone:
  enabled: "-"

  # config is a raw string of default configuration when using a Stateful
  # deployment. Default is to use a PersistentVolumeClaim mounted at /vault/data
  # and store data there. This is only used when using a Replica count of 1, and
  # using a stateful set. This should be HCL.

  # Note: Configuration files are stored in ConfigMaps so sensitive data
  # such as passwords should be either mounted through extraSecretEnvironmentVars
  # or through a Kube secret.  For more information see:
  # https://www.vaultproject.io/docs/platform/k8s/helm/run#protecting-sensitive-vault-configurations
  config: |
    ui = true

    listener "tcp" {
      address = "[::]:8200"
      cluster_address = "[::]:8201"
    }
    storage "file" {
      path = "/vault/data"
    }

    # Example configuration for using auto-unseal, using Google Cloud KMS. The
    # GKMS keys must already exist, and the cluster must have a service account
    # that is authorized to access GCP KMS.
    #seal "gcpckms" {
    #   project     = "vault-helm-dev"
    #   region      = "global"
    #   key_ring    = "vault-helm-unseal-kr"
    #   crypto_key  = "vault-helm-unseal-key"
    #}

# Run Vault in "HA" mode. There are no storage requirements unless audit log
# persistence is required.  In HA mode Vault will configure itself to use Consul
# for its storage backend.  The default configuration provided will work the Consul
# Helm project by default.  It is possible to manually configure Vault to use a
# different HA backend.
ha:
  enabled: true
  replicas: 3

  # Set the api_addr configuration for Vault HA
  # See https://www.vaultproject.io/docs/configuration#api_addr
  # If set to null, this will be set to the Pod IP Address
  apiAddr: null

  # Enables Vault's integrated Raft storage.  Unlike the typical HA modes where
  # Vault's persistence is external (such as Consul), enabling Raft mode will create
  # persistent volumes for Vault to store data according to the configuration under server.dataStorage.
  # The Vault cluster will coordinate leader elections and failovers internally.
  raft:

    # Enables Raft integrated storage
    enabled: false
    # Set the Node Raft ID to the name of the pod
    setNodeId: false

    # Note: Configuration files are stored in ConfigMaps so sensitive data
    # such as passwords should be either mounted through extraSecretEnvironmentVars
    # or through a Kube secret.  For more information see:
    # https://www.vaultproject.io/docs/platform/k8s/helm/run#protecting-sensitive-vault-configurations
    config: |
      ui = true

      listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      storage "raft" {
        path = "/vault/data"
      }

      service_registration "kubernetes" {}

  # config is a raw string of default configuration when using a Stateful
  # deployment. Default is to use a Consul for its HA storage backend.
  # This should be HCL.

  # Note: Configuration files are stored in ConfigMaps so sensitive data
  # such as passwords should be either mounted through extraSecretEnvironmentVars
  # or through a Kube secret.  For more information see:
  # https://www.vaultproject.io/docs/platform/k8s/helm/run#protecting-sensitive-vault-configurations
  config: |
    ui = true

    listener "tcp" {
      address = "[::]:8200"
      cluster_address = "[::]:8201"
      tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
      tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
      tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
    }
    storage "consul" {
      path = "vault"
      address = "HOST_IP:8500"
    }

    telemetry {
      disable_hostname = true
      prometheus_retention_time = "12h"
    }

    service_registration "kubernetes" {}

    # Example configuration for using auto-unseal, using Google Cloud KMS. The
    # GKMS keys must already exist, and the cluster must have a service account
    # that is authorized to access GCP KMS.
    #seal "gcpckms" {
    #   project     = "vault-helm-dev-246514"
    #   region      = "global"
    #   key_ring    = "vault-helm-unseal-kr"
    #   crypto_key  = "vault-helm-unseal-key"
    #}

  # A disruption budget limits the number of pods of a replicated application
  # that are down simultaneously from voluntary disruptions
  disruptionBudget:
    enabled: true

  # maxUnavailable will default to (n/2)-1 where n is the number of
  # replicas. If you'd like a custom value, you can specify an override here.
    maxUnavailable: null

# Definition of the serviceAccount used to run Vault.
# These options are also used when using an external Vault server to validate
# Kubernetes tokens.
serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""
  # Extra annotations for the serviceAccount definition. This can either be
  # YAML or a YAML-formatted multi-line templated string map of the
  # annotations to apply to the serviceAccount.
  annotations: {}

# Settings for the statefulSet used to run Vault.
statefulSet:
  # Extra annotations for the statefulSet. This can either be YAML or a
  # YAML-formatted multi-line templated string map of the annotations to apply
  # to the statefulSet.
  annotations: {}

Vault UI

ui:

True if you want to create a Service entry for the Vault UI.

#
# serviceType can be used to control the type of service created. For
# example, setting this to "LoadBalancer" will create an external load
# balancer (for supported K8S installations) to access the UI.
enabled: true
publishNotReadyAddresses: true
# The service should only contain selectors for active Vault pod
activeVaultPodOnly: true
serviceType: "LoadBalancer"
serviceNodePort: null
externalPort: 32376
targetPort: 8200

# The externalTrafficPolicy can be set to either Cluster or Local
# and is only valid for LoadBalancer and NodePort service types.
# The default value is Cluster.
# ref: https://kubernetes.io/docs/concepts/services-networking/service/#external-traffic-policy
externalTrafficPolicy: Cluster

#loadBalancerSourceRanges:
#   - 10.0.0.0/16
#   - 1.78.23.3/32

# loadBalancerIP:

# Extra annotations to attach to the ui service
# This can either be YAML or a YAML-formatted multi-line templated string map
# of the annotations to apply to the ui service
annotations: {}

secrets-store-csi-driver-provider-vault

csi:

True if you want to install a secrets-store-csi-driver-provider-vault daemonset.

#
# Requires installing the secrets-store-csi-driver separately, see:
# https://github.com/kubernetes-sigs/secrets-store-csi-driver#install-the-secrets-store-csi-driver
#
# With the driver and provider installed, you can mount Vault secrets into volumes
# similar to the Vault Agent injector, and you can also sync those secrets into
# Kubernetes secrets.
enabled: false

image:
  repository: "xxx.xx.x/eccs-vault-dockerimages"
  tag: "vault-csi-provider_0.3.0_0.18.0"
  pullPolicy: IfNotPresent

# volumes is a list of volumes made available to all containers. These are rendered
# via toYaml rather than pre-processed like the extraVolumes value.
# The purpose is to make it easy to share volumes between containers.
volumes: null
# - name: tls
#   secret:
#     secretName: vault-tls

# volumeMounts is a list of volumeMounts for the main server container. These are rendered
# via toYaml rather than pre-processed like the extraVolumes value.
# The purpose is to make it easy to share volumes between containers.
volumeMounts: null
# - name: tls
#   mountPath: "/vault/tls"
#   readOnly: true

# resources: {}
resources:
  requests:
    cpu: 50m
    memory: 128Mi
  limits:
    cpu: 50m
    memory: 128Mi

# Settings for the daemonSet used to run the provider.
daemonSet:
  updateStrategy:
    type: RollingUpdate
    maxUnavailable: ""
  # Extra annotations for the daemonSet. This can either be YAML or a
  # YAML-formatted multi-line templated string map of the annotations to apply
  # to the daemonSet.
  annotations: {}
  # Provider host path (must match the CSI provider's path)
  providersDir: "/etc/kubernetes/secrets-store-csi-providers"
  # Kubelet host path
  kubeletRootDir: "/var/lib/kubelet"

pod:
  # Extra annotations for the provider pods. This can either be YAML or a
  # YAML-formatted multi-line templated string map of the annotations to apply
  # to the pod.
  annotations: {}

  # Toleration Settings for provider pods
  # This should be either a multi-line string or YAML matching the Toleration array
  # in a PodSpec.
  tolerations: []

serviceAccount:
  # Extra annotations for the serviceAccount definition. This can either be
  # YAML or a YAML-formatted multi-line templated string map of the
  # annotations to apply to the serviceAccount.
  annotations: {}

# Used to configure readinessProbe for the pods.
readinessProbe:
  # When a probe fails, Kubernetes will try failureThreshold times before giving up
  failureThreshold: 2
  # Number of seconds after the container has started before probe initiates
  initialDelaySeconds: 5
  # How often (in seconds) to perform the probe
  periodSeconds: 5
  # Minimum consecutive successes for the probe to be considered successful after having failed
  successThreshold: 1
  # Number of seconds after which the probe times out.
  timeoutSeconds: 3
# Used to configure livenessProbe for the pods.
livenessProbe:
  # When a probe fails, Kubernetes will try failureThreshold times before giving up
  failureThreshold: 2
  # Number of seconds after the container has started before probe initiates
  initialDelaySeconds: 5
  # How often (in seconds) to perform the probe
  periodSeconds: 5
  # Minimum consecutive successes for the probe to be considered successful after having failed
  successThreshold: 1
  # Number of seconds after which the probe times out.
  timeoutSeconds: 3

# Enables debug logging.
debug: false

# Pass arbitrary additional arguments to vault-csi-provider.
extraArgs: []

external_certs: enabled: true name: vault-server-tls data: vault_ca: "" vault_crt: "" vault_key: ""

swenson commented 2 years ago

Hi,

It looks like you're using v0.18.0 of the Helm chart. Have you tried with the latest version (v0.20.1 as of right now)? I know we fixed a few bugs around the TLS certificate watcher that produce errors like you mentioned.

anushanagireddy0430 commented 2 years ago

Hi, We have tried the latest version of HelmChart as well. we tested 2 scenarios. please find our observations. Scenario1:

  1. Deployed vault with consul without TLS

Please help us with the scenario and vault with TLS is important for our use-case.

Thanks!

anushanagireddy0430 commented 2 years ago

Hi Team Any update on this bug. Our usecase has stopped. Waiting for update from your side.