Closed derMarcel98 closed 6 months ago
What is your values.yaml
with any sensitive information removed?
Sadly cant upload a yml and when i paste it as code he always formats it into headlines etc..
If you put 3 back ticks before and after it, it will show up as a code block
My bad, used single back ticks. Thanks!
################################################################################
# Global options
################################################################################
global:
image:
# global image pull policy to use for all container images in the chart
# can be overridden by individual image pullPolicy
pullPolicy:
# global list of secret names to use as image pull secrets for all pod specs in the chart
# secrets must exist in the same namespace
# https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
pullSecretNames: []
# global registry to use for all container images in the chart
# can be overridden by individual image registry
registry:
# global labels will be applied to all resources deployed by the chart
labels: {my-label}
################################################################################
# Common options
################################################################################
# override name of the chart
nameOverride:
# override full name of the chart+release
fullnameOverride:
# override the namespace that resources are installed into
namespaceOverride:
# reference a common CA Certificate or Bundle in all nats config `tls` blocks and nats-box contexts
# note: `tls.verify` still must be set in the appropriate nats config `tls` blocks to require mTLS
tlsCA:
enabled: true
# set configMapName in order to mount an existing configMap to dir
configMapName:
# set secretName in order to mount an existing secretName to dir
secretName: nats-certs
# directory to mount the configMap or secret to
dir: /etc/nats-ca-cert
# key in the configMap or secret that contains the CA Certificate or Bundle
key: ca.crt
################################################################################
# NATS Stateful Set and associated resources
################################################################################
############################################################
# NATS config
############################################################
config:
cluster:
enabled: true
port: 6222
# must be 2 or higher when jetstream is enabled
replicas: 3
listen: 0.0.0.0:6222
name: my-cluster-name
# apply to generated route URLs that connect to other pods in the StatefulSet
routeURLs:
# if both user and password are set, they will be added to route URLs
# and the cluster authorization block
user:
password:
# set to true to use FQDN in route URLs
useFQDN: false
k8sClusterDomain: cluster.local
tls:
enabled: true
# set secretName in order to mount an existing secret to dir
secretName: nats-certs
dir: /etc/nats-certs/cluster
cert: tls.crt
key: tls.key
# merge or patch the tls config
# https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls
merge: {}
patch: []
# merge or patch the cluster config
# https://docs.nats.io/running-a-nats-service/configuration/clustering/cluster_config
merge: {
routes: [
nats://my-service.svc.cluster.local:6222,
nats://my-service.svc.cluster.local:6222,
nats://my-service.svc.cluster.local:6222
]
}
patch: []
jetstream:
enabled: true
fileStore:
enabled: true
dir: /data
############################################################
# stateful set -> volume claim templates -> jetstream pvc
############################################################
pvc:
enabled: true
size: 2Gi
storageClassName:
# merge or patch the jetstream pvc
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#persistentvolumeclaim-v1-core
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}-js"
#name:
# defaults to the PVC size
maxSize:
memoryStore:
enabled: false
# ensure that container has a sufficient memory limit greater than maxSize
maxSize: 1Gi
# merge or patch the jetstream config
# https://docs.nats.io/running-a-nats-service/configuration#jetstream
merge: {}
patch: []
nats:
port: 4222
tls:
enabled: true
# set secretName in order to mount an existing secret to dir
secretName: nats-certs
dir: /etc/nats-certs/nats
cert: tls.crt
key: tls.key
# merge or patch the tls config
# https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls
merge: {}
patch: []
leafnodes:
enabled: false
port: 7422
tls:
enabled: false
# set secretName in order to mount an existing secret to dir
secretName:
dir: /etc/nats-certs/leafnodes
cert: tls.crt
key: tls.key
# merge or patch the tls config
# https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls
merge: {}
patch: []
# merge or patch the leafnodes config
# https://docs.nats.io/running-a-nats-service/configuration/leafnodes/leafnode_conf
merge: {}
patch: []
websocket:
enabled: false
port: 8080
tls:
enabled: false
# set secretName in order to mount an existing secret to dir
secretName:
dir: /etc/nats-certs/websocket
cert: tls.crt
key: tls.key
# merge or patch the tls config
# https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls
merge: {}
patch: []
############################################################
# ingress
############################################################
# service must be enabled also
ingress:
enabled: false
# must contain at least 1 host otherwise ingress will not be created
hosts: []
path: /
pathType: Exact
# sets to the ingress class name
className:
# set to an existing secret name to enable TLS on the ingress; applies to all hosts
tlsSecretName:
# merge or patch the ingress
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#ingress-v1-networking-k8s-io
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}-ws"
name:
# merge or patch the websocket config
# https://docs.nats.io/running-a-nats-service/configuration/websocket/websocket_conf
merge: {}
patch: []
mqtt:
enabled: false
port: 1883
tls:
enabled: false
# set secretName in order to mount an existing secret to dir
secretName:
dir: /etc/nats-certs/mqtt
cert: tls.crt
key: tls.key
# merge or patch the tls config
# https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls
merge: {}
patch: []
# merge or patch the mqtt config
# https://docs.nats.io/running-a-nats-service/configuration/mqtt/mqtt_config
merge: {}
patch: []
gateway:
enabled: false
port: 7222
tls:
enabled: false
# set secretName in order to mount an existing secret to dir
secretName:
dir: /etc/nats-certs/gateway
cert: tls.crt
key: tls.key
# merge or patch the tls config
# https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls
merge: {}
patch: []
# merge or patch the gateway config
# https://docs.nats.io/running-a-nats-service/configuration/gateways/gateway#gateway-configuration-block
merge: {}
patch: []
monitor:
enabled: true
port: 8222
tls:
# config.nats.tls must be enabled also
# when enabled, monitoring port will use HTTPS with the options from config.nats.tls
enabled: false
profiling:
enabled: false
port: 65432
resolver:
enabled: false
dir: /data/resolver
############################################################
# stateful set -> volume claim templates -> resolver pvc
############################################################
pvc:
enabled: true
size: 1Gi
storageClassName:
# merge or patch the pvc
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#persistentvolumeclaim-v1-core
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}-resolver"
#name:
# merge or patch the resolver
# https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/jwt/resolver
merge: {}
patch: []
# adds a prefix to the server name, which defaults to the pod name
# helpful for ensuring server name is unique in a super cluster
serverNamePrefix: ""
# merge or patch the nats config
# https://docs.nats.io/running-a-nats-service/configuration
# following special rules apply
# 1. strings that start with << and end with >> will be unquoted
# use this for variables and numbers with units
# 2. keys ending in $include will be switched to include directives
# keys are sorted alphabetically, use prefix before $includes to control includes ordering
# paths should be relative to /etc/nats-config/nats.conf
# example:
#
# merge:
# $include: ./my-config.conf
# zzz$include: ./my-config-last.conf
# server_name: nats
# authorization:
# token: << $TOKEN >>
# jetstream:
# max_memory_store: << 1GB >>
#
# will yield the config:
# {
# include ./my-config.conf;
# "authorization": {
# "token": $TOKEN
# },
# "jetstream": {
# "max_memory_store": 1GB
# },
# "server_name": "nats",
# include ./my-config-last.conf;
# }
merge: {
max_payload: 8388608,
debug: true,
}
patch: []
############################################################
# stateful set -> pod template -> nats container
############################################################
container:
image:
repository: nats
tag: 2.10.5-alpine
pullPolicy:
registry: public.ecr.aws/docker/library
# container port options
# must be enabled in the config section also
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#containerport-v1-core
ports:
nats: {}
leafnodes: {}
websocket: {}
mqtt: {}
cluster: {}
gateway: {}
monitor: {}
profiling: {}
# map with key as env var name, value can be string or map
# example:
#
# env:
# GOMEMLIMIT: 7GiB
# TOKEN:
# valueFrom:
# secretKeyRef:
# name: nats-auth
# key: token
env: {}
# merge or patch the container
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core
merge: {}
patch: []
############################################################
# stateful set -> pod template -> reloader container
############################################################
reloader:
enabled: true
image:
repository: natsio/nats-server-config-reloader
tag: latest
pullPolicy:
registry: private-mirrored-registry
# env var map, see nats.env for an example
env: {}
# all nats container volume mounts with the following prefixes
# will be mounted into the reloader container
natsVolumeMountPrefixes:
- /etc/
# merge or patch the container
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core
merge: {}
patch: []
############################################################
# stateful set -> pod template -> prom-exporter container
############################################################
# config.monitor must be enabled
promExporter:
enabled: true
image:
repository: natsio/prometheus-nats-exporter
tag: latest
pullPolicy:
registry: private-mirrored-registry
port: 7777
# env var map, see nats.env for an example
env: {}
# merge or patch the container
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core
merge: {}
patch: []
############################################################
# prometheus pod monitor
############################################################
podMonitor:
enabled: true
# merge or patch the pod monitor
# https://prometheus-operator.dev/docs/operator/api/#monitoring.coreos.com/v1.PodMonitor
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}"
name:
############################################################
# service
############################################################
service:
enabled: true
# service port options
# additional boolean field enable to control whether port is exposed in the service
# must be enabled in the config section also
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#serviceport-v1-core
ports:
nats:
enabled: true
leafnodes:
enabled: true
websocket:
enabled: true
mqtt:
enabled: true
cluster:
enabled: true
gateway:
enabled: false
monitor:
enabled: true
profiling:
enabled: false
# merge or patch the service
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}"
#name: test
############################################################
# other nats extension points
############################################################
# stateful set
statefulSet:
# merge or patch the stateful set
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#statefulset-v1-apps
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}"
#name:
# stateful set -> pod template
podTemplate:
# adds a hash of the ConfigMap as a pod annotation
# this will cause the StatefulSet to roll when the ConfigMap is updated
configChecksumAnnotation: true
# map of topologyKey: topologySpreadConstraint
# labelSelector will be added to match StatefulSet pods
#
# topologySpreadConstraints:
# kubernetes.io/hostname:
# maxSkew: 1
#
topologySpreadConstraints: {}
# merge or patch the pod template
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#pod-v1-core
merge: {}
patch: []
# headless service
headlessService:
# merge or patch the headless service
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}-headless"
name:
# config map
configMap:
# merge or patch the config map
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#configmap-v1-core
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}-config"
name:
# pod disruption budget
podDisruptionBudget:
enabled: true
# merge or patch the pod disruption budget
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#poddisruptionbudget-v1-policy
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}"
name:
# service account
serviceAccount:
enabled: false
# merge or patch the service account
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#serviceaccount-v1-core
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}"
name:
############################################################
# natsBox
#
# NATS Box Deployment and associated resources
############################################################
natsBox:
enabled: false
############################################################
# NATS contexts
############################################################
contexts:
default:
creds:
# set contents in order to create a secret with the creds file contents
contents:
# set secretName in order to mount an existing secret to dir
secretName:
# defaults to /etc/nats-creds/<context-name>
dir:
key: nats.creds
nkey:
# set contents in order to create a secret with the nkey file contents
contents:
# set secretName in order to mount an existing secret to dir
secretName:
# defaults to /etc/nats-nkeys/<context-name>
dir:
key: nats.nk
# used to connect with client certificates
tls:
# set secretName in order to mount an existing secret to dir
secretName:
# defaults to /etc/nats-certs/<context-name>
dir:
cert: tls.crt
key: tls.key
# merge or patch the context
# https://docs.nats.io/using-nats/nats-tools/nats_cli#nats-contexts
merge: {}
patch: []
# name of context to select by default
defaultContextName: default
############################################################
# deployment -> pod template -> nats-box container
############################################################
container:
image:
repository: natsio/nats-box
tag: 0.14.1
pullPolicy:
registry:
# env var map, see nats.env for an example
env: {}
# merge or patch the container
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core
merge: {}
patch: []
############################################################
# other nats-box extension points
############################################################
# deployment
deployment:
# merge or patch the deployment
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#deployment-v1-apps
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}-box"
name:
# deployment -> pod template
podTemplate:
# merge or patch the pod template
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#pod-v1-core
merge: {}
patch: []
# contexts secret
contextsSecret:
# merge or patch the context secret
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#secret-v1-core
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}-box-contexts"
name:
# contents secret
contentsSecret:
# merge or patch the contents secret
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#secret-v1-core
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}-box-contents"
name:
# service account
serviceAccount:
enabled: false
# merge or patch the service account
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#serviceaccount-v1-core
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}-box"
name:
################################################################################
# Extra user-defined resources
################################################################################
#
# add arbitrary user-generated resources
# example:
#
# config:
# websocket:
# enabled: true
# extraResources:
# - apiVersion: networking.istio.io/v1beta1
# kind: VirtualService
# metadata:
# name:
# $tplYaml: >
# {{ include "nats.fullname" $ | quote }}
# labels:
# $tplYaml: |
# {{ include "nats.labels" $ }}
# spec:
# hosts:
# - demo.nats.io
# gateways:
# - my-gateway
# http:
# - name: default
# match:
# - name: root
# uri:
# exact: /
# route:
# - destination:
# host:
# $tplYaml: >
# {{ .Values.service.name | quote }}
# port:
# number:
# $tplYaml: >
# {{ .Values.config.websocket.port }}
#
extraResources: []
Try getting rid of config.cluster.merge.routes
- that is set automatically for you by the chart
Also -as a note - it is helpful if you don't include the default values in your values.yaml
and only put the values that you are overriding. This has a couple beneficial side effects:
Alright, will do so and post the version here. Thank you
Also -as a note - it is helpful if you don't include the default values in your
values.yaml
and only put the values that you are overriding. This has a couple beneficial side effects:
- It is much easier to see what changes you are making from the default chart values
- If a default value changes in a future release, you won't be overriding it back to the old default
################################################################################
# Global options
################################################################################
global:
labels: {my-label}
# reference a common CA Certificate or Bundle in all nats config `tls` blocks and nats-box contexts
# note: `tls.verify` still must be set in the appropriate nats config `tls` blocks to require mTLS
tlsCA:
enabled: true
# set secretName in order to mount an existing secretName to dir
secretName: nats-certs
# directory to mount the configMap or secret to
dir: /etc/nats-ca-cert
# key in the configMap or secret that contains the CA Certificate or Bundle
key: ca.crt
################################################################################
# NATS Stateful Set and associated resources
################################################################################
############################################################
# NATS config
############################################################
config:
cluster:
enabled: true
port: 6222
# must be 2 or higher when jetstream is enabled
replicas: 3
listen: 0.0.0.0:6222
name: my-cluster
tls:
enabled: true
# set secretName in order to mount an existing secret to dir
secretName: nats-certs
# merge or patch the cluster config
# https://docs.nats.io/running-a-nats-service/configuration/clustering/cluster_config
merge: {
routes: [
nats://my-service.svc.cluster.local:6222,
nats://my-service.svc.cluster.local:6222,
nats://my-service.svc.cluster.local:6222
]
}
jetstream:
enabled: true
fileStore:
enabled: true
dir: /data
############################################################
# stateful set -> volume claim templates -> jetstream pvc
############################################################
pvc:
enabled: true
size: 2Gi
storageClassName:
# merge or patch the jetstream pvc
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#persistentvolumeclaim-v1-core
merge: {}
patch: []
# defaults to "{{ include "nats.fullname" $ }}-js"
#name:
# defaults to the PVC size
maxSize:
memoryStore:
enabled: false
# ensure that container has a sufficient memory limit greater than maxSize
maxSize: 1Gi
# merge or patch the jetstream config
# https://docs.nats.io/running-a-nats-service/configuration#jetstream
merge: {}
patch: []
nats:
port: 4222
tls:
enabled: true
# set secretName in order to mount an existing secret to dir
secretName: nats-certs
merge: {
max_payload: 8388608,
debug: true,
}
############################################################
# stateful set -> pod template -> nats container
############################################################
container:
image:
repository: nats
tag: 2.10.5-alpine
pullPolicy:
registry: public.ecr.aws/docker/library
############################################################
# stateful set -> pod template -> reloader container
############################################################
reloader:
enabled: true
image:
repository: natsio/nats-server-config-reloader
tag: latest
pullPolicy:
registry: private-registry
############################################################
# stateful set -> pod template -> prom-exporter container
############################################################
# config.monitor must be enabled
promExporter:
enabled: true
image:
repository: natsio/prometheus-nats-exporter
tag: latest
pullPolicy:
registry: private-registry
############################################################
# prometheus pod monitor
############################################################
podMonitor:
enabled: true
############################################################
# service
############################################################
service:
enabled: true
# service port options
# additional boolean field enable to control whether port is exposed in the service
# must be enabled in the config section also
# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#serviceport-v1-core
ports:
nats:
enabled: true
leafnodes:
enabled: true
websocket:
enabled: true
mqtt:
enabled: true
cluster:
enabled: true
gateway:
enabled: false
monitor:
enabled: true
profiling:
enabled: false
############################################################
# natsBox
#
# NATS Box Deployment and associated resources
############################################################
natsBox:
enabled: false
extraResources: []
i also faced same problem i think.
values.yaml:
fullnameOverride: nats
namespaceOverride: event
config:
cluster:
enabled: true
port: 6222
replicas: 3
jetstream:
enabled: true
fileStore:
enabled: true
pvc:
enabled: true
storageClassName: resizable
size: 10Gi
nats:
port: 4222
mqtt:
enabled: true
port: 1883
reloader:
enabled: true
UPDATE:
It seems you need to set the cluster name to a different name, as well as each of the servers. So in your values.yaml
you can just set the following, and that will cause the cluster to be named in a unique manner, as well as each of the servers in the cluster.
statefulSet:
name: your-unique-cluster-name
This way you don't need to set prefixes
OLD ANSWER:
Hello! In order to get a distinct server name for each cluster configure a distinct value for config.serverNamePrefix
in each of your values.yaml
that is used for each deployment. This value will be prefixed to your pod name, so it ends up registering server names like :your-prefix_nats-0
config:
serverNamePrefix: 'your-prefix_'
For additional context. In order to follow the trail, these are the hints I used to solve the problem
Check the generated nats.conf
with kubectl get configmap -n nats nats-config -o=yaml
and look at the following bit:
{
...
"server_name": $SERVER_NAME,
...
}
Find how is that SERVER_NAME
env variable is generated.
https://github.com/nats-io/k8s/blob/b572dc26f2c30dddf090c012886c0998a9183d8f/helm/charts/nats/files/stateful-set/nats-container.yaml#L22-L23
And there it is, it is concatenating .Values.config.serverNamePrefix
with the pod name
Happy NATSing!
What version were you using?
Helm-Chart: 1.1.5 Nats-Server: v2.10.5
What environment was the server running in?
On a k8s cluster using the docker images provided from the helm-chart.
Is this defect reproducible?
Yes, use the helm-chart version 1.1.5 and activate cluster with 3 instances, JetStream and tls.
Given the capability you are leveraging, describe your expectation?
I expect the variable server_name to use its described default when the nats.conf is being generated so its unique for each server. Currently is being left empty.
Given the expectation, what is the defect you are observing?
When you apply the chart the server crashes with only a single error message: "nats-server requires server_name to be set". I expect the server_name to be the default value explained in the documentation (i dont set it explicitly) but when I check the generated nats.conf server_name value is blank.