hashicorp / vault-k8s

First-class support for Vault and Kubernetes.
Mozilla Public License 2.0
785 stars 170 forks source link

Vault sidecar agent is not getting added to client application pod #380

Closed pooja20aug closed 2 years ago

pooja20aug commented 2 years ago

Vault with consul is deployed successfully. After that when we try to deploy a sample nginx deployment to test the vault agent injector, vault agent container is not getting added to the pod. The nginx pod displays as 1/1 Running. But is should be 2/2 .

To Reproduce --> vault with consul is deployed successful --> All pods are up and running --> when a sample nginx deployment is deployed to test whether vault injector is able to inject in the other pod. Below issue comes up:
--> This sample deployment comes up as 1/1 Running state but it should be 2/2. Expected output: --> There should be 2 containers for the sample pod (nginx and vault)

Values.yaml of Vault

`cluster_1_deploy: true

Cluster 1 configuration

cluster_1_values:

Available parameters and their default values for the Vault chart.

global:

enabled is the master enabled switch. Setting this to true or false

# will enable or disable all the components within this chart by default.
enabled: true
# Image pull secret to use for registry authentication.
# Alternatively, the value may be specified as an array of strings.
imagePullSecrets:
# imagePullSecrets:
  - name: docker-image
# TLS for end-to-end encrypted transport
tlsDisable: false
# If deploying to OpenShift
openshift: false
# enable monitoring to create dashboard in grafana
monitoring:
  enable: true
  label: "grafana-grafana"
# enable alerting to notify about vault status using prometheus alertmanager
alerting:
  enable: true
  label: "prometheus-prometheus"    
# Create PodSecurityPolicy for pods
psp:
  enable: false
  # Annotation for PodSecurityPolicy.
  # This is a multi-line templated string map, and can also be set as YAML.
  annotations: |
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default,runtime/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName:  runtime/default

injector:

True if you want to enable vault agent injection.

enabled: true

replicas: 1

# Configures the port the injector should listen on
port: 8080

# If multiple replicas are specified, by default a leader will be determined
# so that only one injector attempts to create TLS certificates.
leaderElector:
  enabled: true

# If true, will enable a node exporter metrics endpoint at /metrics.
metrics:
  enabled: false

# External vault server address for the injector to use. Setting this will
# disable deployment of a vault server along with the injector.
externalVaultAddr: ""

# image sets the repo and tag of the vault-k8s image to use for the injector.
image:
  repository: "serodocker.sero.gic.ericsson.se/proj-eccs-dev/eccs-vault-dockerimages"
  tag: "vault-k8s_0.14.1_0.18.0"
  pullPolicy: IfNotPresent

# agentImage sets the repo and tag of the Vault image to use for the Vault Agent
# containers.  This should be set to the official Vault image.  Vault 1.3.1+ is
# required.
agentImage:
  repository: "serodocker.sero.gic.ericsson.se/proj-eccs-dev/eccs-vault-dockerimages"
  tag: "vault_1.9.0_0.18.0"

# The default values for the injected Vault Agent containers.
agentDefaults:
  # For more information on configuring resources, see the K8s documentation:
  # https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
  cpuLimit: "500m"
  cpuRequest: "250m"
  memLimit: "128Mi"
  memRequest: "64Mi"

  # Default template type for secrets when no custom template is specified.
  # Possible values include: "json" and "map".
  template: "map"

  # Default values within Agent's template_config stanza.
  templateConfig:
    exitOnRetryFailure: true
    staticSecretRenderInterval: ""

# Mount Path of the Vault Kubernetes Auth Method.
authPath: "auth/kubernetes"

# Configures the log verbosity of the injector.
# Supported log levels include: trace, debug, info, warn, error
logLevel: "info"

# Configures the log format of the injector. Supported log formats: "standard", "json".
logFormat: "standard"

# Configures all Vault Agent sidecars to revoke their token when shutting down
revokeOnShutdown: false

# namespaceSelector is the selector for restricting the webhook to only
# specific namespaces.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector
# for more details.
# Example:
# namespaceSelector:
#    matchLabels:
#      sidecar-injector: enabled
namespaceSelector: {}
# objectSelector is the selector for restricting the webhook to only
# specific labels.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-objectselector
# for more details.
# Example:
# objectSelector:
#    matchLabels:
#      vault-sidecar-injector: enabled
objectSelector: {}

# Configures failurePolicy of the webhook. The "unspecified" default behaviour deoends on the
# API Version of the WebHook.
# To block pod creation while webhook is unavailable, set the policy to `Fail` below.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#failure-policy
#
failurePolicy: Ignore

# Extra annotations to attach to the webhook
webhookAnnotations: {}

certs:
  # secretName is the name of the secret that has the TLS certificate and
  # private key to serve the injector webhook. If this is null, then the
  # injector will default to its automatic management mode that will assign
  # a service account to the injector to generate its own certificates.
  secretName: vault-server-tls

  # caBundle is a base64-encoded PEM-encoded certificate bundle for the CA
  # that signed the TLS certificate that the webhook serves. This must be set
  # if secretName is non-null, unless an external service like cert-manager is
  # keeping the caBundle updated.
  caBundle: ""

  # certName and keyName are the names of the files within the secret for
  # the TLS cert and private key, respectively. These have reasonable
  # defaults but can be customized if necessary.
  certName: vault.crt
  keyName: vault.key

resources:
  requests:
    memory: 256Mi
    cpu: 250m
  limits:
    memory: 256Mi
    cpu: 250m 

# extraEnvironmentVars is a list of extra environment variables to set in the
# injector deployment.
extraEnvironmentVars: {}
  # KUBERNETES_SERVICE_HOST: kubernetes.default.svc

# Affinity Settings for injector pods
# This can either be multi-line string or YAML matching the PodSpec's affinity field.
# Commenting out or setting as empty the affinity variable, will allow
# deployment of multiple replicas to single node services such as Minikube.
affinity: |
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: {{ template "vault.name" . }}-agent-injector
            app.kubernetes.io/instance: "{{ .Release.Name }}"
            component: webhook
        topologyKey: kubernetes.io/hostname

# Toleration Settings for injector pods
# This should be either a multi-line string or YAML matching the Toleration array
# in a PodSpec.
tolerations: []

# nodeSelector labels for server pod assignment, formatted as a multi-line string or YAML map.
# ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
# Example:
# nodeSelector:
#   beta.kubernetes.io/arch: amd64
nodeSelector: {}

# Priority class for injector pods
priorityClassName: ""

# Extra annotations to attach to the injector pods
# This can either be YAML or a YAML-formatted multi-line templated string map
# of the annotations to apply to the injector pods
annotations: {}

# Extra labels to attach to the agent-injector
# This should be a YAML map of the labels to apply to the injector
extraLabels: {}

# Should the injector pods run on the host network (useful when using
# an alternate CNI in EKS)
hostNetwork: false

# Injector service specific config
service:
  # Extra annotations to attach to the injector service
  annotations: {}

# A disruption budget limits the number of pods of a replicated application
# that are down simultaneously from voluntary disruptions
podDisruptionBudget: {}
# podDisruptionBudget:
#   maxUnavailable: 1

server:

If not set to true, Vault server will not be installed. See vault.mode in _helpers.tpl for implementation details

enabled: true

# [Enterprise Only] This value refers to a Kubernetes secret that you have
# created that contains your enterprise license. If you are not using an
# enterprise image or if you plan to introduce the license key via another
# route, then leave secretName blank ("") or set it to null.
# Requires Vault Enterprise 1.8 or later.
enterpriseLicense:
  # The name of the Kubernetes secret that holds the enterprise license. The
  # secret must be in the same namespace that Vault is installed into.
  secretName: ""
  # The key within the Kubernetes secret that holds the enterprise license.
  secretKey: "license"

# Resource requests, limits, etc. for the server cluster placement. This
# should map directly to the value of the resources field for a PodSpec.
# By default no direct resource request is made.

image:
  repository: "serodocker.sero.gic.ericsson.se/proj-eccs-dev/eccs-vault-dockerimages"
  tag: "vault_1.9.0_0.18.0"
  # Overrides the default Image Pull Policy
  pullPolicy: IfNotPresent

# Configure the Update Strategy Type for the StatefulSet
# See https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
updateStrategyType: "OnDelete"

# Configure the logging verbosity for the Vault server.
# Supported log levels include: trace, debug, info, warn, error
logLevel: ""

# Configure the logging format for the Vault server.
# Supported log formats include: standard, json
logFormat: ""

resources:
  requests:
    memory: 256Mi
    cpu: 250m
  limits:
    memory: 256Mi
    cpu: 250m

# Ingress allows ingress services to be created to allow external access
# from Kubernetes to access Vault pods.
# If deployment is on OpenShift, the following block is ignored.
# In order to expose the service, use the route section below
ingress:
  enabled: false
  labels: {}
    # traffic: external
  annotations: {}
    # |
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
    #   or
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"

  # Optionally use ingressClassName instead of deprecated annotation.
  # See: https://kubernetes.io/docs/concepts/services-networking/ingress/#deprecated-annotation
  ingressClassName: ""

  # As of Kubernetes 1.19, all Ingress Paths must have a pathType configured. The default value below should be sufficient in most cases.
  # See: https://kubernetes.io/docs/concepts/services-networking/ingress/#path-types for other possible values.
  pathType: Prefix

  # When HA mode is enabled and K8s service registration is being used,
  # configure the ingress to point to the Vault active service.
  activeService: true
  hosts:
    - host: chart-example.local
      paths: []
  ## Extra paths to prepend to the host configuration. This is useful when working with annotation based services.
  extraPaths: []
  # - path: /*
  #   backend:
  #     service:
  #       name: ssl-redirect
  #       port:
  #         number: use-annotation
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

# OpenShift only - create a route to expose the service
# The created route will be of type passthrough
route:
  enabled: false

  # When HA mode is enabled and K8s service registration is being used,
  # configure the route to point to the Vault active service.
  activeService: true

  labels: {}
  annotations: {}
  host: chart-example.local

# authDelegator enables a cluster role binding to be attached to the service
# account.  This cluster role binding can be used to setup Kubernetes auth
# method.  https://www.vaultproject.io/docs/auth/kubernetes.html
authDelegator:
  enabled: true

# extraInitContainers is a list of init containers. Specified as a YAML list.
# This is useful if you need to run a script to provision TLS certificates or
# write out configuration files in a dynamic way.
extraInitContainers: null
  # # This example installs a plugin pulled from github into the /usr/local/libexec/vault/oauthapp folder,
  # # which is defined in the volumes value.
  # - name: oauthapp
  #   image: "alpine"
  #   command: [sh, -c]
  #   args:
  #     - cd /tmp &&
  #       wget https://github.com/puppetlabs/vault-plugin-secrets-oauthapp/releases/download/v1.2.0/vault-plugin-secrets-oauthapp-v1.2.0-linux-amd64.tar.xz -O oauthapp.xz &&
  #       tar -xf oauthapp.xz &&
  #       mv vault-plugin-secrets-oauthapp-v1.2.0-linux-amd64 /usr/local/libexec/vault/oauthapp &&
  #       chmod +x /usr/local/libexec/vault/oauthapp
  #   volumeMounts:
  #     - name: plugins
  #       mountPath: /usr/local/libexec/vault

# extraContainers is a list of sidecar containers. Specified as a YAML list.
extraContainers: null

# shareProcessNamespace enables process namespace sharing between Vault and the extraContainers
# This is useful if Vault must be signaled, e.g. to send a SIGHUP for log rotation
shareProcessNamespace: false

# extraArgs is a string containing additional Vault server arguments.
extraArgs: ""

# Used to define custom readinessProbe settings
readinessProbe:
  enabled: true
  # If you need to use a http path instead of the default exec
  # path: /v1/sys/health?standbyok=true

  # When a probe fails, Kubernetes will try failureThreshold times before giving up
  failureThreshold: 2
  # Number of seconds after the container has started before probe initiates
  initialDelaySeconds: 5
  # How often (in seconds) to perform the probe
  periodSeconds: 5
  # Minimum consecutive successes for the probe to be considered successful after having failed
  successThreshold: 1
  # Number of seconds after which the probe times out.
  timeoutSeconds: 3
# Used to enable a livenessProbe for the pods
livenessProbe:
  enabled: false
  path: "/v1/sys/health?standbyok=true"
  # When a probe fails, Kubernetes will try failureThreshold times before giving up
  failureThreshold: 2
  # Number of seconds after the container has started before probe initiates
  initialDelaySeconds: 60
  # How often (in seconds) to perform the probe
  periodSeconds: 5
  # Minimum consecutive successes for the probe to be considered successful after having failed
  successThreshold: 1
  # Number of seconds after which the probe times out.
  timeoutSeconds: 3

# Optional duration in seconds the pod needs to terminate gracefully.
# See: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
terminationGracePeriodSeconds: 10

# Used to set the sleep time during the preStop step
preStopSleepSeconds: 5

# Used to define commands to run after the pod is ready.
# This can be used to automate processes such as initialization
# or boostrapping auth methods.
postStart: []
# - /bin/sh
# - -c
# - /vault/userconfig/myscript/run.sh

# extraEnvironmentVars is a list of extra environment variables to set with the stateful set. These could be
# used to include variables required for auto-unseal.
extraEnvironmentVars:
  VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
  # GOOGLE_REGION: global
  # GOOGLE_PROJECT: myproject
  # GOOGLE_APPLICATION_CREDENTIALS: /vault/userconfig/myproject/myproject-creds.json

# extraSecretEnvironmentVars is a list of extra environment variables to set with the stateful set.
# These variables take value from existing Secret objects.
extraSecretEnvironmentVars: []
  # - envName: AWS_SECRET_ACCESS_KEY
  #   secretName: vault
  #   secretKey: AWS_SECRET_ACCESS_KEY

# Deprecated: please use 'volumes' instead.
# extraVolumes is a list of extra volumes to mount. These will be exposed
# to Vault in the path `/vault/userconfig/<name>/`. The value below is
# an array of objects, examples are shown below.
extraVolumes:
  - type: secret
    name: vault-server-tls
  # - type: secret (or "configMap")
  #   name: my-secret
  #   path: null # default is `/vault/userconfig`

# volumes is a list of volumes made available to all containers. These are rendered
# via toYaml rather than pre-processed like the extraVolumes value.
# The purpose is to make it easy to share volumes between containers.
volumes: null
#   - name: plugins
#     emptyDir: {}

# volumeMounts is a list of volumeMounts for the main server container. These are rendered
# via toYaml rather than pre-processed like the extraVolumes value.
# The purpose is to make it easy to share volumes between containers.
volumeMounts: null
#   - mountPath: /usr/local/libexec/vault
#     name: plugins
#     readOnly: true

# Affinity Settings
# Commenting out or setting as empty the affinity variable, will allow
# deployment to single node services such as Minikube
# This should be either a multi-line string or YAML matching the PodSpec's affinity field.
affinity: |
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: {{ template "vault.name" . }}
            app.kubernetes.io/instance: "{{ .Release.Name }}"
            component: server
        topologyKey: kubernetes.io/hostname

# Toleration Settings for server pods
# This should be either a multi-line string or YAML matching the Toleration array
# in a PodSpec.
tolerations: []

# nodeSelector labels for server pod assignment, formatted as a multi-line string or YAML map.
# ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
# Example:
# nodeSelector:
#   beta.kubernetes.io/arch: amd64
nodeSelector: {}

# Enables network policy for server pods
networkPolicy:
  enabled: false
  egress: []
  # egress:
  # - to:
  #   - ipBlock:
  #       cidr: 10.0.0.0/24
  #   ports:
  #   - protocol: TCP
  #     port: 443

# Priority class for server pods
priorityClassName: ""

# Extra labels to attach to the server pods
# This should be a YAML map of the labels to apply to the server pods
extraLabels: {}

# Extra annotations to attach to the server pods
# This can either be YAML or a YAML-formatted multi-line templated string map
# of the annotations to apply to the server pods
annotations: {}

# Enables a headless service to be used by the Vault Statefulset
service:
  enabled: true
  # clusterIP controls whether a Cluster IP address is attached to the
  # Vault service within Kubernetes.  By default the Vault service will
  # be given a Cluster IP address, set to None to disable.  When disabled
  # Kubernetes will create a "headless" service.  Headless services can be
  # used to communicate with pods directly through DNS instead of a round robin
  # load balancer.
  # clusterIP: None

  # Configures the service type for the main Vault service.  Can be ClusterIP
  # or NodePort.
  #type: ClusterIP

  # The externalTrafficPolicy can be set to either Cluster or Local
  # and is only valid for LoadBalancer and NodePort service types.
  # The default value is Cluster.
  # ref: https://kubernetes.io/docs/concepts/services-networking/service/#external-traffic-policy
  externalTrafficPolicy: Cluster

  # If type is set to "NodePort", a specific nodePort value can be configured,
  # will be random if left blank.
  #nodePort: 30000

  # Port on which Vault server is listening
  port: 8200
  # Target port to which the service should be mapped to
  targetPort: 8200
  # Extra annotations for the service definition. This can either be YAML or a
  # YAML-formatted multi-line templated string map of the annotations to apply
  # to the service.
  annotations: {}

  #annotations:
    #service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    #service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    #service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-03b3e92ca9c1108f3, subnet-0ae9557ecdb9de70b
    #service.beta.kubernetes.io/aws-load-balancer-type: external
  #loadBalancerIP: {}
  #port: 30075
  #type: LoadBalancer

# This configures the Vault Statefulset to create a PVC for data
# storage when using the file or raft backend storage engines.
# See https://www.vaultproject.io/docs/configuration/storage/index.html to know more
dataStorage:
  enabled: true
  # Size of the PVC created
  size: 10Gi
  # Location where the PVC will be mounted.
  mountPath: "/vault/data"
  # Name of the storage class to use.  If null it will use the
  # configured default Storage Class.
  storageClass: null
  # Access Mode of the storage device being used for the PVC
  accessMode: ReadWriteOnce
  # Annotations to apply to the PVC
  annotations: {}

# This configures the Vault Statefulset to create a PVC for audit
# logs.  Once Vault is deployed, initialized and unseal, Vault must
# be configured to use this for audit logs.  This will be mounted to
# /vault/audit
# See https://www.vaultproject.io/docs/audit/index.html to know more
auditStorage:
  enabled: false
  # Size of the PVC created
  size: 10Gi
  # Location where the PVC will be mounted.
  mountPath: "/vault/audit"
  # Name of the storage class to use.  If null it will use the
  # configured default Storage Class.
  storageClass: null
  # Access Mode of the storage device being used for the PVC
  accessMode: ReadWriteOnce
  # Annotations to apply to the PVC
  annotations: {}

# Run Vault in "dev" mode. This requires no further setup, no state management,
# and no initialization. This is useful for experimenting with Vault without
# needing to unseal, store keys, et. al. All data is lost on restart - do not
# use dev mode for anything other than experimenting.
# See https://www.vaultproject.io/docs/concepts/dev-server.html to know more
dev:
  enabled: false

  # Set VAULT_DEV_ROOT_TOKEN_ID value
  devRootToken: "root"

# Run Vault in "standalone" mode. This is the default mode that will deploy if
# no arguments are given to helm. This requires a PVC for data storage to use
# the "file" backend.  This mode is not highly available and should not be scaled
# past a single replica.
standalone:
  enabled: "-"

  # config is a raw string of default configuration when using a Stateful
  # deployment. Default is to use a PersistentVolumeClaim mounted at /vault/data
  # and store data there. This is only used when using a Replica count of 1, and
  # using a stateful set. This should be HCL.

  # Note: Configuration files are stored in ConfigMaps so sensitive data
  # such as passwords should be either mounted through extraSecretEnvironmentVars
  # or through a Kube secret.  For more information see:
  # https://www.vaultproject.io/docs/platform/k8s/helm/run#protecting-sensitive-vault-configurations
  config: |
    ui = true

    listener "tcp" {
      address = "[::]:8200"
      cluster_address = "[::]:8201"
    }
    storage "file" {
      path = "/vault/data"
    }

    # Example configuration for using auto-unseal, using Google Cloud KMS. The
    # GKMS keys must already exist, and the cluster must have a service account
    # that is authorized to access GCP KMS.
    #seal "gcpckms" {
    #   project     = "vault-helm-dev"
    #   region      = "global"
    #   key_ring    = "vault-helm-unseal-kr"
    #   crypto_key  = "vault-helm-unseal-key"
    #}

# Run Vault in "HA" mode. There are no storage requirements unless audit log
# persistence is required.  In HA mode Vault will configure itself to use Consul
# for its storage backend.  The default configuration provided will work the Consul
# Helm project by default.  It is possible to manually configure Vault to use a
# different HA backend.
ha:
  enabled: true
  replicas: 3

  # Set the api_addr configuration for Vault HA
  # See https://www.vaultproject.io/docs/configuration#api_addr
  # If set to null, this will be set to the Pod IP Address
  apiAddr: null

  # Enables Vault's integrated Raft storage.  Unlike the typical HA modes where
  # Vault's persistence is external (such as Consul), enabling Raft mode will create
  # persistent volumes for Vault to store data according to the configuration under server.dataStorage.
  # The Vault cluster will coordinate leader elections and failovers internally.
  raft:

    # Enables Raft integrated storage
    enabled: false
    # Set the Node Raft ID to the name of the pod
    setNodeId: false

    # Note: Configuration files are stored in ConfigMaps so sensitive data
    # such as passwords should be either mounted through extraSecretEnvironmentVars
    # or through a Kube secret.  For more information see:
    # https://www.vaultproject.io/docs/platform/k8s/helm/run#protecting-sensitive-vault-configurations
    config: |
      ui = true

      listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      storage "raft" {
        path = "/vault/data"
      }

      service_registration "kubernetes" {}

  # config is a raw string of default configuration when using a Stateful
  # deployment. Default is to use a Consul for its HA storage backend.
  # This should be HCL.

  # Note: Configuration files are stored in ConfigMaps so sensitive data
  # such as passwords should be either mounted through extraSecretEnvironmentVars
  # or through a Kube secret.  For more information see:
  # https://www.vaultproject.io/docs/platform/k8s/helm/run#protecting-sensitive-vault-configurations
  config: |
    ui = true

    listener "tcp" {
      address = "[::]:8200"
      cluster_address = "[::]:8201"
      tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
      tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
      tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
    }
    storage "consul" {
      path = "vault"
      address = "HOST_IP:8500"
    }

    telemetry {
      disable_hostname = true
      prometheus_retention_time = "12h"
    }

    service_registration "kubernetes" {}

    # Example configuration for using auto-unseal, using Google Cloud KMS. The
    # GKMS keys must already exist, and the cluster must have a service account
    # that is authorized to access GCP KMS.
    #seal "gcpckms" {
    #   project     = "vault-helm-dev-246514"
    #   region      = "global"
    #   key_ring    = "vault-helm-unseal-kr"
    #   crypto_key  = "vault-helm-unseal-key"
    #}

  # A disruption budget limits the number of pods of a replicated application
  # that are down simultaneously from voluntary disruptions
  disruptionBudget:
    enabled: true

  # maxUnavailable will default to (n/2)-1 where n is the number of
  # replicas. If you'd like a custom value, you can specify an override here.
    maxUnavailable: null

# Definition of the serviceAccount used to run Vault.
# These options are also used when using an external Vault server to validate
# Kubernetes tokens.
serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""
  # Extra annotations for the serviceAccount definition. This can either be
  # YAML or a YAML-formatted multi-line templated string map of the
  # annotations to apply to the serviceAccount.
  annotations: {}

# Settings for the statefulSet used to run Vault.
statefulSet:
  # Extra annotations for the statefulSet. This can either be YAML or a
  # YAML-formatted multi-line templated string map of the annotations to apply
  # to the statefulSet.
  annotations: {}

Vault UI

ui:

True if you want to create a Service entry for the Vault UI.

#
# serviceType can be used to control the type of service created. For
# example, setting this to "LoadBalancer" will create an external load
# balancer (for supported K8S installations) to access the UI.
enabled: true
publishNotReadyAddresses: true
# The service should only contain selectors for active Vault pod
activeVaultPodOnly: true
serviceType: "LoadBalancer"
serviceNodePort: null
externalPort: 32376
targetPort: 8200

# The externalTrafficPolicy can be set to either Cluster or Local
# and is only valid for LoadBalancer and NodePort service types.
# The default value is Cluster.
# ref: https://kubernetes.io/docs/concepts/services-networking/service/#external-traffic-policy
externalTrafficPolicy: Cluster

#loadBalancerSourceRanges:
#   - 10.0.0.0/16
#   - 1.78.23.3/32

# loadBalancerIP:

# Extra annotations to attach to the ui service
# This can either be YAML or a YAML-formatted multi-line templated string map
# of the annotations to apply to the ui service
annotations: {}

secrets-store-csi-driver-provider-vault

csi:

True if you want to install a secrets-store-csi-driver-provider-vault daemonset.

#
# Requires installing the secrets-store-csi-driver separately, see:
# https://github.com/kubernetes-sigs/secrets-store-csi-driver#install-the-secrets-store-csi-driver
#
# With the driver and provider installed, you can mount Vault secrets into volumes
# similar to the Vault Agent injector, and you can also sync those secrets into
# Kubernetes secrets.
enabled: false

image:
  repository: "serodocker.sero.gic.ericsson.se/proj-eccs-dev/eccs-vault-dockerimages"
  tag: "vault-csi-provider_0.3.0_0.18.0"
  pullPolicy: IfNotPresent

# volumes is a list of volumes made available to all containers. These are rendered
# via toYaml rather than pre-processed like the extraVolumes value.
# The purpose is to make it easy to share volumes between containers.
volumes: null
# - name: tls
#   secret:
#     secretName: vault-tls

# volumeMounts is a list of volumeMounts for the main server container. These are rendered
# via toYaml rather than pre-processed like the extraVolumes value.
# The purpose is to make it easy to share volumes between containers.
volumeMounts: null
# - name: tls
#   mountPath: "/vault/tls"
#   readOnly: true

# resources: {}
resources:
  requests:
    cpu: 50m
    memory: 128Mi
  limits:
    cpu: 50m
    memory: 128Mi

# Settings for the daemonSet used to run the provider.
daemonSet:
  updateStrategy:
    type: RollingUpdate
    maxUnavailable: ""
  # Extra annotations for the daemonSet. This can either be YAML or a
  # YAML-formatted multi-line templated string map of the annotations to apply
  # to the daemonSet.
  annotations: {}
  # Provider host path (must match the CSI provider's path)
  providersDir: "/etc/kubernetes/secrets-store-csi-providers"
  # Kubelet host path
  kubeletRootDir: "/var/lib/kubelet"

pod:
  # Extra annotations for the provider pods. This can either be YAML or a
  # YAML-formatted multi-line templated string map of the annotations to apply
  # to the pod.
  annotations: {}

  # Toleration Settings for provider pods
  # This should be either a multi-line string or YAML matching the Toleration array
  # in a PodSpec.
  tolerations: []

serviceAccount:
  # Extra annotations for the serviceAccount definition. This can either be
  # YAML or a YAML-formatted multi-line templated string map of the
  # annotations to apply to the serviceAccount.
  annotations: {}

# Used to configure readinessProbe for the pods.
readinessProbe:
  # When a probe fails, Kubernetes will try failureThreshold times before giving up
  failureThreshold: 2
  # Number of seconds after the container has started before probe initiates
  initialDelaySeconds: 5
  # How often (in seconds) to perform the probe
  periodSeconds: 5
  # Minimum consecutive successes for the probe to be considered successful after having failed
  successThreshold: 1
  # Number of seconds after which the probe times out.
  timeoutSeconds: 3
# Used to configure livenessProbe for the pods.
livenessProbe:
  # When a probe fails, Kubernetes will try failureThreshold times before giving up
  failureThreshold: 2
  # Number of seconds after the container has started before probe initiates
  initialDelaySeconds: 5
  # How often (in seconds) to perform the probe
  periodSeconds: 5
  # Minimum consecutive successes for the probe to be considered successful after having failed
  successThreshold: 1
  # Number of seconds after which the probe times out.
  timeoutSeconds: 3

# Enables debug logging.
debug: false

# Pass arbitrary additional arguments to vault-csi-provider.
extraArgs: []

external_certs: enabled: true name: vault-server-tls data: vault_ca: "" vault_crt: "" vault_key: ""

`

sample nginx deployment.yaml: `--- apiVersion: apps/v1 kind: Deployment metadata: name: app-example-deployment55 spec: replicas: 1 selector: matchLabels: app: app-example55 template: metadata: labels: app: app-example55 annotations: vault.hashicorp.com/agent-inject: 'true' vault.hashicorp.com/agent-inject-status: "update" vault.hashicorp.com/agent-cache-enable: "true" vault.hashicorp.com/agent-cache-use-auto-auth-token: "true" vault.hashicorp.com/agent-inject-token: "true" vault.hashicorp.com/log-level: "debug" vault.hashicorp.com/agent-inject-command-db-creds: "sh -c 'kill -HUP $(pidof client)'" vault.hashicorp.com/agent-configmap: 'my-configmap'

vault.hashicorp.com/ca-cert: '/vault/tls/ca.crt'

    #vault.hashicorp.com/client-cert: '/vault/tls/client.crt'
    #vault.hashicorp.com/client-key: '/vault/tls/client.key'
    vault.hashicorp.com/tls-secret: 'vault-server-tls'
spec:
  containers:
    - name: app
      image: nginx
      resources:
        limits:
          cpu: 250m
          memory: 256Mi
        requests:
          cpu: 250m
          memory: 256Mi
  imagePullSecrets:
  - name: docker-image

apiVersion: v1 kind: ConfigMap metadata: name: my-configmap data: config.hcl: | "auto_auth" = { "method" = { "config" = { "role" = "eopsrole" } "type" = "kubernetes" }

  "sink" = {
    "config" = {
      "path" = "/home/vault/.token"
    }

    "type" = "file"
  }
}

"exit_after_auth" = false
"pid_file" = "/home/vault/.pid"

"template" = {
  "contents" = "{{- with secret \"secret-v2/nsc\" -}}username---{{ .Data.data.user }} password ---- {{ .Data.data.password }}{{- end }}"
  "destination" = "/vault/secrets/db-creds"
}

"vault" = {
  "address" = "https://vaultcluster1ha-eccs-vault-helmcharts:8200"
  "ca_cert" = "/vault/tls/vault.ca"
  "client_cert" = "/vault/tls/vault.crt"
  "client_key" = "/vault/tls/vault.key"
}

config-init.hcl: | "auto_auth" = { "method" = { "config" = { "role" = "eopsrole" } "type" = "kubernetes" }

  "sink" = {
    "config" = {
      "path" = "/home/vault/.token"
    }

    "type" = "file"
  }
}

"exit_after_auth" = true
"pid_file" = "/home/vault/.pid"

"template" = {
  "contents" = "{{- with secret \"secret-v2/nsc\" -}}username---{{ .Data.data.user }} password ---- {{ .Data.data.password }}{{- end }}"
  "destination" = "/vault/secrets/db-creds"
}

"vault" = {
  "address" = "https://vaultcluster1ha-eccs-vault-helmcharts:8200"
  "ca_cert" = "/vault/tls/vault.ca"
  "client_cert" = "/vault/tls/vault.crt"
  "client_key" = "/vault/tls/vault.key"
}

`

Expected behavior Sample deployment pod should come up with 2/2 containers

Environment

image

tomhjp commented 2 years ago

I'm afraid there's too much noise here to help you debug the issue. The best place to debug deployment issues like this is the discuss forums: https://discuss.hashicorp.com/c/vault/30. I would recommend that you just include the helm chart values that you have changed from the default, make sure you use markdown triple-backticks so that it's more easily readable, and include as many commands as you can to help others reproduce the issue. Sorry I can't be of more help here!