Closed shuiyunwudi closed 2 years ago
t-eckert
commented
2 years ago Hi @shuiyunwudi,
Thank you for your question. The reason you are getting this error is Kubernetes doesn't have the CRDs installed for Consul. The fix for this is to upgrade to a version of Consul on Kubernetes which supports CRDs. This documentation will shed some light on what's needed to install CRDs: https://www.consul.io/docs/k8s/crds
t-eckert
commented
2 years ago It's worth noting that this requires Consul 1.8.4 or greater.
shuiyunwudi
commented
2 years ago @t-eckert thank you for your answer, I will try to upgrade the version of consul and other components.
shuiyunwudi
commented
2 years ago Now, I upgrade to Consul-helm 0.28.0 version, consul 1.8.4 version and consul-k8s 0.22.0. when I deploy it, the pods' status like that:
[mcloud@XX_111_1_155 consul-helm-0.28.0]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
chartmuseum-d57777b5-nsq6j 1/1 Running 0 136d
consul-6b6rk 0/1 Running 0 20m
consul-btv8k 0/1 Running 0 20m
consul-connect-injector-webhook-deployment-7d769b5789-26c9n 1/1 Running 0 20m
consul-controller-6cd9f9d8c5-vlhxq 1/1 Running 0 20m
consul-mesh-gateway-7c8954bbb9-m62z4 0/2 Init:CrashLoopBackOff 8 20m
consul-qczwp 0/1 Running 0 20m
consul-server-0 0/1 Pending 0 20m
consul-webhook-cert-manager-667b4f5c8-sgkvw 1/1 Running 0 20mand the log for mesh-gateway pod:
[mcloud@XX_111_1_155 consul-helm-0.28.0]$ kubectl logs -f consul-mesh-gateway-7c8954bbb9-m62z4 -c service-init
Address "XX.96.183.110" written to /tmp/address.txt successfully
Error registering service "mesh-gateway": Unexpected response code: 500 (could not retrieve initial service_defaults config for service "mesh-gateway": No known Consul servers)
I apply custome proxy-defaults.yaml file ,i will attch it below. Anyone knows why and how to solve it?
Here is proxy-defaults.yaml:
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
spec:
meshGateway:
mode: localBelow is the consul-helm values.yaml:
global:
enabled: true
name: null
domain: consul
image: "consul:1.8.4"
imagePullSecrets: []
imageK8S: "hashicorp/consul-k8s:0.22.0"
datacenter: dev-consul-connect-dc2
enablePodSecurityPolicies: false
gossipEncryption:
secretName: ""
secretKey: ""
tls:
enabled: true
enableAutoEncrypt: false
serverAdditionalDNSSANs: []
serverAdditionalIPSANs: []
verify: false
httpsOnly: false
caCert:
secretName: consul-federation
secretKey: caCert
caKey:
secretName: consul-federation
secretKey: caKey
enableConsulNamespaces: false
acls:
manageSystemACLs: false
bootstrapToken:
secretName: null
secretKey: null
createReplicationToken: false
replicationToken:
# The name of the Kubernetes secret.
secretName: null
# The key of the Kubernetes secret.
secretKey: null
# Configure federation.
federation:
enabled: true
createFederationSecret: false
lifecycleSidecarContainer:
resources:
requests:
memory: "25Mi"
cpu: "20m"
limits:
memory: "50Mi"
cpu: "20m"
imageEnvoy: "envoyproxy/envoy-alpine:v1.14.7"
openshift:
# If true, the Helm chart will create necessary configuration for running
# its components on OpenShift.
enabled: false
server:
enabled: true
image: null
replicas: 1
bootstrapExpect: 1
enterpriseLicense:
secretName: null
secretKey: null
exposeGossipAndRPCPorts: false
ports:
serflan:
port: 8301
storage: 10Gi
storageClass: null
connect: true
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "100Mi"
cpu: "100m"
securityContext:
runAsNonRoot: true
runAsGroup: 1000
runAsUser: 100
fsGroup: 1000
updatePartition: 0
disruptionBudget:
enabled: true
maxUnavailable: null
extraConfig: |
{
"log_level":"DEBUG",
"primary_detacenter":"dev-consul-connect",
"primary_gateways":["xx.xxx.xx.xxx","xxx.xxx.xx.xxx"]
}
extraVolumes:
- type: secret
name: consul-federation
items:
- key: serverConfigJSON
path: config.json
load: true
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: {{ template "consul.name" . }}
release: "{{ .Release.Name }}"
component: server
topologyKey: kubernetes.io/hostname
tolerations: ""
priorityClassName: ""
extraLabels: null
annotations: null
# Server service properties.
service:
annotations: null
extraEnvironmentVars: {}
externalServers:
enabled: false
hosts: []
httpsPort: 8501
# The server name to use as the SNI host header when connecting with HTTPS.
# @type: string
tlsServerName: null
useSystemRoots: false
k8sAuthMethodHost: null
# Values that configure running a Consul client on Kubernetes nodes.
client:
enabled: true
image: null
join: null
dataDirectoryHostPath: null
grpc: true
exposeGossipPorts: false
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "100Mi"
cpu: "100m"
securityContext:
runAsNonRoot: true
runAsGroup: 1000
runAsUser: 100
fsGroup: 1000
extraConfig: |
{}
extraVolumes: []
tolerations: ""
nodeSelector: null
affinity: {}
priorityClassName: ""
annotations: null
extraEnvironmentVars: {}
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
updateStrategy: null
snapshotAgent:
# If true, the chart will install resources necessary to run the snapshot agent.
enabled: false
# The number of snapshot agents to run.
replicas: 2
configSecret:
# The name of the Kubernetes secret.
secretName: null
# The key of the Kubernetes secret.
secretKey: null
# Resource settings for snapshot agent pods.
resources:
requests:
memory: "50Mi"
cpu: "50m"
limits:
memory: "50Mi"
cpu: "50m"
caCert: null
dns:
# @type: boolean
enabled: true
# Used to control the type of service created. For
# example, setting this to "LoadBalancer" will create an external load
# balancer (for supported K8S installations)
type: ClusterIP
# Set a predefined cluster IP for the DNS service.
# Useful if you need to reference the DNS service's IP
# address in CoreDNS config.
# @type: string
clusterIP: null
# Extra annotations to attach to the dns service
# This should be a multi-line string of
# annotations to apply to the dns Service
# @type: string
annotations: null
# Additional ServiceSpec values
# This should be a multi-line string mapping directly to a Kubernetes
# ServiceSpec object.
# @type: string
additionalSpec: null
# Values that configure the Consul UI.
ui:
enabled: "-"
# Configure the service for the Consul UI.
service:
enabled: true
# The service type to register.
# @type: string
type: LoadBalancer
# Annotations to apply to the UI service.
#
# Example:
#
# ```yaml
# annotations: |
# 'annotation-key': annotation-value
# ```
# @type: string
annotations: null
# Additional ServiceSpec values
# This should be a multi-line string mapping directly to a Kubernetes
# ServiceSpec object.
# @type: string
additionalSpec: null
syncCatalog:
enabled: false
image: null
default: true
priorityClassName: ""
toConsul: true
toK8S: true
k8sPrefix: ""
k8sAllowNamespaces: ["test"]
k8sDenyNamespaces: ["kube-system", "kube-public"]
k8sSourceNamespace: null
consulNamespaces:
consulDestinationNamespace: "default"
mirroringK8S: true
mirroringK8SPrefix: "myconsul-"
addK8SNamespaceSuffix: true
consulPrefix: "fromk8s-"
k8sTag: null
consulNodeName: "k8s-sync"
# Syncs services of the ClusterIP type, which may
# or may not be broadly accessible depending on your Kubernetes cluster.
# Set this to false to skip syncing ClusterIP services.
syncClusterIPServices: true
nodePortSyncType: ExternalFirst
aclSyncToken:
# The name of the Kubernetes secret.
secretName: null
# The key of the Kubernetes secret.
secretKey: null
nodeSelector: null
affinity: null
tolerations: null
resources:
requests:
memory: "50Mi"
cpu: "50m"
limits:
memory: "50Mi"
cpu: "50m"
logLevel: debug
consulWriteInterval: null
connectInject:
enabled: true
image: null
default: true
healthChecks:
enabled: true
reconcilePeriod: "1m"
envoyExtraArgs: "-- -l off --component-log-level upstream:trace,http:trace,router:trace,config:debug "
# Optional priorityClassName.
priorityClassName: ""
# The Docker image for Consul to use when performing Connect injection.
# Defaults to global.image.
# @type: string
imageConsul: null
# Log verbosity level. One of "debug", "info", "warn", or "error".
logLevel: info
# Resource settings for connect inject pods.
resources:
requests:
memory: "50Mi"
cpu: "50m"
limits:
memory: "50Mi"
cpu: "50m"
namespaceSelector: null
k8sAllowNamespaces: ["test"]
k8sDenyNamespaces: ["mock"]
consulNamespaces:
consulDestinationNamespace: "default"
mirroringK8S: false
mirroringK8SPrefix: ""
certs:
secretName: null
caBundle: ""
certName: tls.crt
keyName: tls.key
nodeSelector: null
affinity: null
tolerations: null
aclBindingRuleSelector: "serviceaccount.name!=default"
overrideAuthMethodName: ""
aclInjectToken:
secretName: null
secretKey: null
centralConfig:
enabled: true
defaultProtocol: http
proxyDefaults: |
{}
sidecarProxy:
resources:
requests:
memory: null
# Recommended default: 100m
# @type: string
cpu: null
limits:
# Recommended default: 100Mi
# @type: string
memory: null
# Recommended default: 100m
# @type: string
cpu: null
# Resource settings for the Connect injected init container.
initContainer:
resources:
requests:
memory: "25Mi"
cpu: "50m"
limits:
memory: "150Mi"
cpu: "50m"
# Controller handles config entry custom resources.
# Requires consul >= 1.8.4.
# ServiceIntentions require consul 1.9+.
controller:
enabled: true
replicas: 1
# Log verbosity level. One of "debug", "info", "warn", or "error".
logLevel: debug
# Resource settings for controller pods.
resources:
limits:
cpu: 100m
memory: 50Mi
requests:
cpu: 100m
memory: 50Mi
# Optional YAML string to specify a nodeSelector config.
# @type: string
nodeSelector: null
# Optional YAML string to specify tolerations.
# @type: string
tolerations: null
# Affinity Settings
# This should be a multi-line string matching the affinity object
# @type: string
affinity: null
# Optional priorityClassName.
priorityClassName: ""
# Mesh Gateways enable Consul Connect to work across Consul datacenters.
meshGateway:
enabled: true
globalMode: local
# Number of replicas for the Deployment.
replicas: 1
# What gets registered as WAN address for the gateway.
wanAddress:
source: "Service"
port: 443
static: ""
# The service option configures the Service that fronts the Gateway Deployment.
service:
# Whether to create a Service or not.
enabled: true
# Type of service, ex. LoadBalancer, ClusterIP.
type: LoadBalancer
port: 443
nodePort: null
annotations: null
# Optional YAML string that will be appended to the Service spec.
# @type: string
additionalSpec: null
# If set to true, gateway Pods will run on the host network.
hostNetwork: false
dnsPolicy: null
consulServiceName: "mesh-gateway"
# Port that the gateway will run on inside the container.
containerPort: 8443
hostPort: null
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "100Mi"
cpu: "100m"
# Resource settings for the `copy-consul-bin` init container.
initCopyConsulContainer:
resources:
requests:
memory: "25Mi"
cpu: "50m"
limits:
memory: "150Mi"
cpu: "50m"
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: {{ template "consul.name" . }}
release: "{{ .Release.Name }}"
component: mesh-gateway
topologyKey: kubernetes.io/hostname
# Optional YAML string to specify tolerations.
# @type: string
tolerations: null
# Optional YAML string to specify a nodeSelector config.
# @type: string
nodeSelector: null
# Optional priorityClassName.
priorityClassName: ""
# Annotations to apply to the mesh gateway deployment.
#
# Example:
#
# ```yaml
# annotations: |
# 'annotation-key': annotation-value
# ```
# @type: string
annotations: null
ingressGateways:
# Enable ingress gateway deployment. Requires `connectInject.enabled=true`
# and `client.enabled=true`.
enabled: false
# Defaults sets default values for all gateway fields. With the exception
# of annotations, defining any of these values in the `gateways` list
# will override the default values provided here. Annotations will
# include both the default annotations and any additional ones defined
# for a specific gateway.
defaults:
# Number of replicas for each ingress gateway defined.
replicas: 2
# The service options configure the Service that fronts the gateway Deployment.
service:
# Type of service: LoadBalancer, ClusterIP or NodePort. If using NodePort service
# type, you must set the desired nodePorts in the `ports` setting below.
type: ClusterIP
# Ports that will be exposed on the service and gateway container. Any
# ports defined as ingress listeners on the gateway's Consul configuration
# entry should be included here. The first port will be used as part of
# the Consul service registration for the gateway and be listed in its
# SRV record. If using a NodePort service type, you must specify the
# desired nodePort for each exposed port.
# @type: array<map>
# @default: [{port: 8080, port: 8443}]
# @recurse: false
ports:
- port: 8080
nodePort: null
- port: 8443
nodePort: null
# Annotations to apply to the ingress gateway service. Annotations defined
# here will be applied to all ingress gateway services in addition to any
# service annotations defined for a specific gateway in `ingressGateways.gateways`.
#
# Example:
#
# ```yaml
# annotations: |
# 'annotation-key': annotation-value
# ```
# @type: string
annotations: null
# Optional YAML string that will be appended to the Service spec.
# @type: string
additionalSpec: null
# Resource limits for all ingress gateway pods
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "100Mi"
cpu: "100m"
# Resource settings for the `copy-consul-bin` init container.
initCopyConsulContainer:
resources:
requests:
memory: "25Mi"
cpu: "50m"
limits:
memory: "150Mi"
cpu: "50m"
# By default, we set an anti-affinity so that two of the same gateway pods
# won't be on the same node. NOTE: Gateways require that Consul client agents are
# also running on the nodes alongside each gateway pod.
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: {{ template "consul.name" . }}
release: "{{ .Release.Name }}"
component: ingress-gateway
topologyKey: kubernetes.io/hostname
# Optional YAML string to specify tolerations.
# @type: string
tolerations: null
# Optional YAML string to specify a nodeSelector config.
# @type: string
nodeSelector: null
# Optional priorityClassName.
priorityClassName: ""
# Annotations to apply to the ingress gateway deployment. Annotations defined
# here will be applied to all ingress gateway deployments in addition to any
# annotations defined for a specific gateway in `ingressGateways.gateways`.
#
# Example:
#
# ```yaml
# annotations: |
# "annotation-key": 'annotation-value'
# ```
# @type: string
annotations: null
# [Enterprise Only] `consulNamespace` defines the Consul namespace to register
# the gateway into. Requires `global.enableConsulNamespaces` to be true and
# Consul Enterprise v1.7+ with a valid Consul Enterprise license.
# Note: The Consul namespace MUST exist before the gateway is deployed.
consulNamespace: "default"
# Gateways is a list of gateway objects. The only required field for
# each is `name`, though they can also contain any of the fields in
# `defaults`. Values defined here override the defaults except in the
# case of annotations where both will be applied.
# @type: array<map>
gateways:
- name: ingress-gateway
# Configuration options for terminating gateways. Default values for all
# terminating gateways are defined in `terminatingGateways.defaults`. Any of
# these values may be overridden in `terminatingGateways.gateways` for a
# specific gateway with the exception of annotations. Annotations will
# include both the default annotations and any additional ones defined
# for a specific gateway.
# Requirements: consul >= 1.8.0 and consul-k8s >= 0.16.0 if using
# global.acls.manageSystemACLs and consul-k8s >= 0.10.0 if not.
terminatingGateways:
# Enable terminating gateway deployment. Requires `connectInject.enabled=true`
# and `client.enabled=true`.
enabled: false
# Defaults sets default values for all gateway fields. With the exception
# of annotations, defining any of these values in the `gateways` list
# will override the default values provided here. Annotations will
# include both the default annotations and any additional ones defined
# for a specific gateway.
defaults:
# Number of replicas for each terminating gateway defined.
replicas: 2
# A list of extra volumes to mount. These will be exposed to Consul in the path `/consul/userconfig/<name>/`.
#
# Example:
#
# ```yaml
# extraVolumes:
# - type: secret
# name: my-secret
# items: # optional items array
# - key: key
# path: path # secret will now mount to /consul/userconfig/my-secret/path
# ```
# @type: array<map>
extraVolumes: []
# Resource limits for all terminating gateway pods
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "100Mi"
cpu: "100m"
# Resource settings for the `copy-consul-bin` init container.
initCopyConsulContainer:
resources:
requests:
memory: "25Mi"
cpu: "50m"
limits:
memory: "150Mi"
cpu: "50m"
# By default, we set an anti-affinity so that two of the same gateway pods
# won't be on the same node. NOTE: Gateways require that Consul client agents are
# also running on the nodes alongside each gateway pod.
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: {{ template "consul.name" . }}
release: "{{ .Release.Name }}"
component: terminating-gateway
topologyKey: kubernetes.io/hostname
# Optional YAML string to specify tolerations.
# @type: string
tolerations: null
# Optional YAML string to specify a nodeSelector config.
# @type: string
nodeSelector: null
# Optional priorityClassName.
# @type: string
priorityClassName: ""
# Annotations to apply to the terminating gateway deployment. Annotations defined
# here will be applied to all terminating gateway deployments in addition to any
# annotations defined for a specific gateway in `terminatingGateways.gateways`.
#
# Example:
#
# ```yaml
# annotations: |
# 'annotation-key': annotation-value
# ```
# @type: string
annotations: null
# [Enterprise Only] `consulNamespace` defines the Consul namespace to register
# the gateway into. Requires `global.enableConsulNamespaces` to be true and
# Consul Enterprise v1.7+ with a valid Consul Enterprise license.
# Note: The Consul namespace MUST exist before the gateway is deployed.
consulNamespace: "default"
# Gateways is a list of gateway objects. The only required field for
# each is `name`, though they can also contain any of the fields in
# `defaults`. Values defined here override the defaults except in the
# case of annotations where both will be applied.
# @type: array<map>
gateways:
- name: terminating-gateway
# Control whether a test Pod manifest is generated when running helm template.
# When using helm install, the test Pod is not submitted to the cluster so this
# is only useful when running helm template.
tests:
enabled: trueThe second problem was that pv for consul need to be created manually before execute command "kubectl apply -f values.yaml" to deploy consul-helm.
Question
when i follow the reference multi-cluster proxydefaults to apply proxy defaults, it show error like that:
however, the reference point out apiVersion - Must be set to consul.hashicorp.com/v1alpha1, proxy-defaults reference
Anyone know the reason and how to solve it?
CLI Commands (consul-k8s, consul-k8s-control-plane, helm)
Helm Configuration
proxy-defaults.yaml:
Logs
Current understanding and Expected behavior
Environment details
consul-helm: 0.26.0; consul: 1.8.3; consul-k8s: 0.20.0
Additional Context