search

nats-io / k8s

NATS on Kubernetes with Helm Charts
Apache License 2.0
446 stars 302 forks source link

Upgrading from 0.x to 1.0 according to docs does not work #820

Closed stemid closed 10 months ago

stemid commented 10 months ago

What version were you using?

2.9.20

What environment was the server running in?

k8s

Is this defect reproducible?

  1. Follow UPGRADING.md for nats-1.0.3.

Given the capability you are leveraging, describe your expectation?

helm template \
  --version "1.0.3" \
  -f patched-values.yaml \
  -s templates/stateful-set.yaml \
  nats \
  nats/nats | grep -A2 volumeClaimTemplates
  volumeClaimTemplates:
  - metadata:
      name: nats-js-pvc

Here is my patched-values.yaml file.

################################################################################
# Global options
################################################################################
global:
  image:
    # global image pull policy to use for all container images in the chart
    # can be overridden by individual image pullPolicy
    pullPolicy:
    # global registry to use for all container images in the chart
    # can be overridden by individual image registry
    registry:

  # global labels will be applied to all resources deployed by the chart
  labels: {}

################################################################################
# Common options
################################################################################
# override name of the chart
nameOverride:
# override full name of the chart+release
fullnameOverride:
# override the namespace that resources are installed into
namespaceOverride:

# reference a common CA Certificate or Bundle in all nats config `tls` blocks and nats-box contexts
# note: `tls.verify` still must be set in the appropriate nats config `tls` blocks to require mTLS
tlsCA:
  enabled: false
  # set configMapName in order to mount an existing configMap to dir
  configMapName:
  # set secretName in order to mount an existing secretName to dir
  secretName:
  # directory to mount the configMap or secret to
  dir: /etc/nats-ca-cert
  # key in the configMap or secret that contains the CA Certificate or Bundle
  key: ca.crt

################################################################################
# NATS Stateful Set and associated resources
################################################################################

############################################################
# NATS config
############################################################
config:
  cluster:
    enabled: true
    port: 6222
    # must be 2 or higher when jetstream is enabled
    replicas: 3

    # apply to generated route URLs that connect to other pods in the StatefulSet
    routeURLs:
      # if both user and password are set, they will be added to route URLs
      # and the cluster authorization block
      user:
      password:
      # set to true to use FQDN in route URLs
      useFQDN: false
      k8sClusterDomain: cluster.local

    tls:
      enabled: false
      # set secretName in order to mount an existing secret to dir
      secretName:
      dir: /etc/nats-certs/cluster
      cert: tls.crt
      key: tls.key
      # merge or patch the tls config
      # https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls
      merge: {}
      patch: []

    # merge or patch the cluster config
    # https://docs.nats.io/running-a-nats-service/configuration/clustering/cluster_config
    merge: {}
    patch: []

  jetstream:
    enabled: true

    fileStore:
      enabled: true
      dir: /data

      ############################################################
      # stateful set -> volume claim templates -> jetstream pvc
      ############################################################
      pvc:
        name:
          $tplYaml: >-
            {{ include "nats.fullname" . }}-js-pvc
        enabled: true
        size: 2Gi
        storageClassName: rook-ceph-block

        # merge or patch the jetstream pvc
        # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#persistentvolumeclaim-v1-core
        merge: {}
        patch: []
        # defaults to "{{ include "nats.fullname" $ }}-js"
        name:

      # defaults to the PVC size
      maxSize:

    memoryStore:
      enabled: false
      # ensure that container has a sufficient memory limit greater than maxSize
      maxSize: 1Gi

    # merge or patch the jetstream config
    # https://docs.nats.io/running-a-nats-service/configuration#jetstream
    merge: {}
    patch: []

  nats:
    port: 4222
    tls:
      enabled: false
      # set secretName in order to mount an existing secret to dir
      secretName:
      dir: /etc/nats-certs/nats
      cert: tls.crt
      key: tls.key
      # merge or patch the tls config
      # https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls
      merge: {}
      patch: []

  leafnodes:
    enabled: false
    port: 7422
    tls:
      enabled: false
      # set secretName in order to mount an existing secret to dir
      secretName:
      dir: /etc/nats-certs/leafnodes
      cert: tls.crt
      key: tls.key
      # merge or patch the tls config
      # https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls
      merge: {}
      patch: []

    # merge or patch the leafnodes config
    # https://docs.nats.io/running-a-nats-service/configuration/leafnodes/leafnode_conf
    merge: {}
    patch: []

  websocket:
    enabled: false
    port: 8080
    tls:
      enabled: false
      # set secretName in order to mount an existing secret to dir
      secretName:
      dir: /etc/nats-certs/websocket
      cert: tls.crt
      key: tls.key
      # merge or patch the tls config
      # https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls
      merge: {}
      patch: []

    ############################################################
    # ingress
    ############################################################
    # service must be enabled also
    ingress:
      enabled: false
      # must contain at least 1 host otherwise ingress will not be created
      hosts: []
      path: /
      pathType: Exact
      # sets to the ingress class name
      className:
      # set to an existing secret name to enable TLS on the ingress; applies to all hosts
      tlsSecretName:

      # merge or patch the ingress
      # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#ingress-v1-networking-k8s-io
      merge: {}
      patch: []
      # defaults to "{{ include "nats.fullname" $ }}-ws"
      name:

    # merge or patch the websocket config
    # https://docs.nats.io/running-a-nats-service/configuration/websocket/websocket_conf
    merge: {}
    patch: []

  mqtt:
    enabled: false
    port: 1883
    tls:
      enabled: false
      # set secretName in order to mount an existing secret to dir
      secretName:
      dir: /etc/nats-certs/mqtt
      cert: tls.crt
      key: tls.key
      # merge or patch the tls config
      # https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls
      merge: {}
      patch: []

    # merge or patch the mqtt config
    # https://docs.nats.io/running-a-nats-service/configuration/mqtt/mqtt_config
    merge: {}
    patch: []

  gateway:
    enabled: false
    port: 7222
    tls:
      enabled: false
      # set secretName in order to mount an existing secret to dir
      secretName:
      dir: /etc/nats-certs/gateway
      cert: tls.crt
      key: tls.key
      # merge or patch the tls config
      # https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls
      merge: {}
      patch: []

    # merge or patch the gateway config
    # https://docs.nats.io/running-a-nats-service/configuration/gateways/gateway#gateway-configuration-block
    merge: {}
    patch: []

  monitor:
    enabled: true
    port: 8222
    tls:
      # config.nats.tls must be enabled also
      # when enabled, monitoring port will use HTTPS with the options from config.nats.tls
      enabled: false

  profiling:
    enabled: false
    port: 65432

  resolver:
    enabled: false
    dir: /data/resolver

    ############################################################
    # stateful set -> volume claim templates -> resolver pvc
    ############################################################
    pvc:
      enabled: true
      size: 1Gi
      storageClassName:

      # merge or patch the pvc
      # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#persistentvolumeclaim-v1-core
      merge: {}
      patch: []
      # defaults to "{{ include "nats.fullname" $ }}-resolver"
      name:

    # merge or patch the resolver
    # https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/jwt/resolver
    merge: {}
    patch: []

  # adds a prefix to the server name, which defaults to the pod name
  # helpful for ensuring server name is unique in a super cluster
  serverNamePrefix: ""

  # merge or patch the nats config
  # https://docs.nats.io/running-a-nats-service/configuration
  # following special rules apply
  #  1. strings that start with << and end with >> will be unquoted
  #     use this for variables and numbers with units
  #  2. keys ending in $include will be switched to include directives
  #     keys are sorted alphabetically, use prefix before $includes to control includes ordering
  #     paths should be relative to /etc/nats-config/nats.conf
  # example:
  #
  #   merge:
  #     $include: ./my-config.conf
  #     zzz$include: ./my-config-last.conf
  #     server_name: nats
  #     authorization:
  #       token: << $TOKEN >>
  #     jetstream:
  #       max_memory_store: << 1GB >>
  #
  # will yield the config:
  # {
  #   include ./my-config.conf;
  #   "authorization": {
  #     "token": $TOKEN
  #   },
  #   "jetstream": {
  #     "max_memory_store": 1GB
  #   },
  #   "server_name": "nats",
  #   include ./my-config-last.conf;
  # }
  merge: {}
  patch: []

############################################################
# stateful set -> pod template -> nats container
############################################################
container:
  image:
    repository: nats
    tag: 2.9.22-alpine
    pullPolicy:
    registry:

  # container port options
  # must be enabled in the config section also
  # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#containerport-v1-core
  ports:
    nats: {}
    leafnodes: {}
    websocket: {}
    mqtt: {}
    cluster: {}
    gateway: {}
    monitor: {}
    profiling: {}

  # map with key as env var name, value can be string or map
  # example:
  #
  #   env:
  #     GOMEMLIMIT: 7GiB
  #     TOKEN:
  #       valueFrom:
  #         secretKeyRef:
  #           name: nats-auth
  #           key: token
  env: {}

  # merge or patch the container
  # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core
  merge: {}
  patch: []

############################################################
# stateful set -> pod template -> reloader container
############################################################
reloader:
  enabled: true
  image:
    repository: natsio/nats-server-config-reloader
    tag: 0.11.0
    pullPolicy:
    registry:

  # env var map, see nats.env for an example
  env: {}

  # all nats container volume mounts with the following prefixes
  # will be mounted into the reloader container
  natsVolumeMountPrefixes:
  - /etc/

  # merge or patch the container
  # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core
  merge: {}
  patch: []

############################################################
# stateful set -> pod template -> prom-exporter container
############################################################
# config.monitor must be enabled
promExporter:
  enabled: false
  image:
    repository: natsio/prometheus-nats-exporter
    tag: 0.12.0
    pullPolicy:
    registry:

  port: 7777
  # env var map, see nats.env for an example
  env: {}

  # merge or patch the container
  # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core
  merge: {}
  patch: []

  ############################################################
  # prometheus pod monitor
  ############################################################
  podMonitor:
    enabled: false

    # merge or patch the pod monitor
    # https://prometheus-operator.dev/docs/operator/api/#monitoring.coreos.com/v1.PodMonitor
    merge: {}
    patch: []
    # defaults to "{{ include "nats.fullname" $ }}"
    name:

############################################################
# service
############################################################
service:
  enabled: true

  # service port options
  # additional boolean field enable to control whether port is exposed in the service
  # must be enabled in the config section also
  # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#serviceport-v1-core
  ports:
    nats:
      enabled: true
    leafnodes:
      enabled: true
    websocket:
      enabled: true
    mqtt:
      enabled: true
    cluster:
      enabled: false
    gateway:
      enabled: false
    monitor:
      enabled: false
    profiling:
      enabled: false

  # merge or patch the service
  # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core
  merge: {}
  patch: []
  # defaults to "{{ include "nats.fullname" $ }}"
  name:
    $tplYaml: >-
      {{ include "nats.fullname" . }}-svc

############################################################
# other nats extension points
############################################################

# stateful set
statefulSet:
  patch:
    - op: remove
      path: /spec/selector/matchLabels/app.kubernetes.io~1component
    - $tplYamlSpread: |-
        {{- if and 
          .Values.config.jetstream.enabled
          .Values.config.jetstream.fileStore.enabled
          .Values.config.jetstream.fileStore.pvc.enabled
          .Values.config.resolver.enabled
          .Values.config.resolver.pvc.enabled
        }}
        - op: move
          from: /spec/volumeClaimTemplates/0
          path: /spec/volumeClaimTemplates/1
        {{- else}}
        []
        {{- end }}
  # merge or patch the stateful set
  # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#statefulset-v1-apps
  merge: {}
  patch: []
  # defaults to "{{ include "nats.fullname" $ }}"
  name:

# stateful set -> pod template
podTemplate:
  # adds a hash of the ConfigMap as a pod annotation
  # this will cause the StatefulSet to roll when the ConfigMap is updated
  configChecksumAnnotation: true

  # map of topologyKey: topologySpreadConstraint
  # labelSelector will be added to match StatefulSet pods
  #
  # topologySpreadConstraints:
  #   kubernetes.io/hostname:
  #     maxSkew: 1
  #
  topologySpreadConstraints: {}

  # merge or patch the pod template
  # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#pod-v1-core
  merge: {}
  patch: []

# headless service
headlessService:
  # merge or patch the headless service
  # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core
  merge: {}
  patch: []
  name:
    $tplYaml: >-
      {{ include "nats.fullname" . }}

# config map
configMap:
  # merge or patch the config map
  # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#configmap-v1-core
  merge: {}
  patch: []
  # defaults to "{{ include "nats.fullname" $ }}-config"
  name:

# pod disruption budget
podDisruptionBudget:
  enabled: true
  # merge or patch the pod disruption budget
  # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#poddisruptionbudget-v1-policy
  merge: {}
  patch: []
  # defaults to "{{ include "nats.fullname" $ }}"
  name:

# service account
serviceAccount:
  enabled: true
  # merge or patch the service account
  # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#serviceaccount-v1-core
  merge: {}
  patch: []
  # defaults to "{{ include "nats.fullname" $ }}"
  name:

############################################################
# natsBox
#
# NATS Box Deployment and associated resources
############################################################
natsBox:
  enabled: true

  ############################################################
  # NATS contexts
  ############################################################
  contexts:
    default:
      creds:
        # set contents in order to create a secret with the creds file contents
        contents:
        # set secretName in order to mount an existing secret to dir
        secretName:
        # defaults to /etc/nats-creds/<context-name>
        dir:
        key: nats.creds
      nkey:
        # set contents in order to create a secret with the nkey file contents
        contents:
        # set secretName in order to mount an existing secret to dir
        secretName:
        # defaults to /etc/nats-nkeys/<context-name>
        dir:
        key: nats.nk
      # used to connect with client certificates
      tls:
        # set secretName in order to mount an existing secret to dir
        secretName:
        # defaults to /etc/nats-certs/<context-name>
        dir:
        cert: tls.crt
        key: tls.key

      # merge or patch the context
      # https://docs.nats.io/using-nats/nats-tools/nats_cli#nats-contexts
      merge: {}
      patch: []

  # name of context to select by default
  defaultContextName: default

  ############################################################
  # deployment -> pod template -> nats-box container
  ############################################################
  container:
    image:
      repository: natsio/nats-box
      tag: 0.13.9
      pullPolicy:
      registry:

    # env var map, see nats.env for an example
    env: {}

    # merge or patch the container
    # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core
    merge: {}
    patch: []

  ############################################################
  # other nats-box extension points
  ############################################################

  # deployment
  deployment:
    patch:
      - op: replace
        path: /spec/selector/matchLabels
        value:
          app: nats-box
      - op: add
        path: /spec/template/metadata/labels/app
        value: nats-box
    # merge or patch the deployment
    # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#deployment-v1-apps
    merge: {}
    patch: []
    # defaults to "{{ include "nats.fullname" $ }}-box"
    name:

  # deployment -> pod template
  podTemplate:
    # merge or patch the pod template
    # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#pod-v1-core
    merge: {}
    patch: []

  # contexts secret
  contextsSecret:
    # merge or patch the context secret
    # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#secret-v1-core
    merge: {}
    patch: []
    # defaults to "{{ include "nats.fullname" $ }}-box-contexts"
    name:

  # contents secret
  contentsSecret:
    # merge or patch the contents secret
    # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#secret-v1-core
    merge: {}
    patch: []
    # defaults to "{{ include "nats.fullname" $ }}-box-contents"
    name:

  # service account
  serviceAccount:
    enabled: false
    # merge or patch the service account
    # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#serviceaccount-v1-core
    merge: {}
    patch: []
    # defaults to "{{ include "nats.fullname" $ }}-box"
    name:

################################################################################
# Extra user-defined resources
################################################################################
#
# add arbitrary user-generated resources
# example:
#
# config:
#   websocket:
#     enabled: true
# extraResources:
# - apiVersion: networking.istio.io/v1beta1
#   kind: VirtualService
#   metadata:
#     name:
#       $tplYaml: >
#         {{ include "nats.fullname" $ | quote }}
#     labels:
#       $tplYaml: |
#         {{ include "nats.labels" $ }}
#   spec:
#     hosts:
#     - demo.nats.io
#     gateways:
#     - my-gateway
#     http:
#     - name: default
#       match:
#       - name: root
#         uri:
#           exact: /
#       route:
#       - destination:
#           host:
#             $tplYaml: >
#               {{ .Values.service.name | quote }}
#           port:
#             number:
#               $tplYaml: >
#                 {{ .Values.config.websocket.port }}
#
extraResources: []

Given the expectation, what is the defect you are observing?

helm template -n nats --version "1.0.3" -f patched-values.yaml -s templates/stateful-set.yaml nats nats/nats|grep -A2 volumeClaim
  volumeClaimTemplates:
  - metadata:
      name: nats-js

This creates new PVC's with names like nats-js-nats-0 instead of re-using the old PVC with names like nats-js-pvc-nats-0.

I also tried setting the name without a $tplYaml hardcoded like name: nats-js-pvc but I still get nats-js as name in the resulting output.

stemid commented 10 months ago

Ok I can close this issue. Sorry but I just noticed when reading through the text that there is a second name: statement under mine, it overrides mine probably.