search

hashicorp / consul-helm

Helm chart to install Consul and other associated components.
Mozilla Public License 2.0
419 stars 386 forks source link

Chart Documentation #300

Closed bsakweson closed 4 years ago

bsakweson commented 4 years ago

Would be best to have a proper documentation on how this chart can be successful executed? I am talking about configuration documentation here not how to use helm. Consul as a service mesh itself is pretty engaging, not having a proper documentation makes it even more difficult. I would not mind to write one but I have not been able to successful run this on my cluster without errors. Let me explain:

The first thing I did was to go to the consul documentation and follow individual components document to set up cluster but still not able to get something that I am comfortable with to run even in preprod environment for testing.

  1. After I read the official consul documentation and made sure I had a clear understanding of what all those different components do, I went ahead and cranked the chart with default values and enabled all components. Here is my configuration file: 
    
# Available parameters and their default values for the Consul chart.

global:

enabled is the master enabled switch. Setting this to true or false

will enable or disable all the components within this chart by default.

Each component can be overridden using the component-specific "enabled"

value.

enabled: true

Domain to register the Consul DNS server to listen for.

domain: consul

Image is the name (and tag) of the Consul Docker image for clients and

servers below. This can be overridden per component.

#

Examples:

image: "consul:1.5.0"

image: "hashicorp/consul-enterprise:1.5.0-ent" # Enterprise Consul image

image: "consul:1.6.1"

imageK8S is the name (and tag) of the consul-k8s Docker image that

is used for functionality such as the catalog sync. This can be overridden

per component below.

Note: support for the catalog sync's liveness and readiness probes was added

to consul-k8s v0.6.0. If using an older consul-k8s version, you may need to

remove these checks to make the sync work.

If using mesh gateways and bootstrapACLs then must be >= 0.9.0.

imageK8S: "hashicorp/consul-k8s:0.9.4"

Datacenter is the name of the datacenter that the agents should register

as. This shouldn't be changed once the Consul cluster is up and running

since Consul doesn't support an automatic way to change this value

currently: https://github.com/hashicorp/consul/issues/1858

datacenter: dc1

enablePodSecurityPolicies is a boolean flag that controls whether pod

security policies are created for the consul components created by this

chart. See https://kubernetes.io/docs/concepts/policy/pod-security-policy/

enablePodSecurityPolicies: false

Gossip encryption key. To enable gossip encryption, provide the name of

a Kubernetes secret that contains a gossip key. You can create a gossip

key with the "consul keygen" command.

See https://www.consul.io/docs/commands/keygen.html

gossipEncryption: secretName: null secretKey: null

bootstrapACLs will automatically create and assign ACL tokens within

the Consul cluster. This currently requires enabling both servers and

clients within Kubernetes. Additionally requires Consul v1.4+ and

consul-k8s v0.8.0+.

bootstrapACLs: false

Server, when enabled, configures a server cluster to run. This should

be disabled if you plan on connecting to a Consul cluster external to

the Kube cluster.

server: enabled: "-" image: null replicas: 3 bootstrapExpect: 3 # Should <= replicas count

enterpriseLicense refers to a Kubernetes secret that you have created that

contains your enterprise license. It is required if you are using an

enterprise binary. Defining it here applies it to your cluster once a leader

has been elected. If you are not using an enterprise image

or if you plan to introduce the license key via another route, then set

these fields to null.

enterpriseLicense: secretName: null secretKey: null

storage and storageClass are the settings for configuring stateful

storage for the server pods. storage should be set to the disk size of

the attached volume. storageClass is the class of storage which defaults

to null (the Kube cluster will pick the default).

storage: 10Gi storageClass: null

connect will enable Connect on all the servers, initializing a CA

for Connect-related connections. Other customizations can be done

via the extraConfig setting.

connect: true

Resource requests, limits, etc. for the server cluster placement. This

should map directly to the value of the resources field for a PodSpec,

formatted as a multi-line string. By default no direct resource request

is made.

resources: null

updatePartition is used to control a careful rolling update of Consul

servers. This should be done particularly when changing the version

of Consul. Please refer to the documentation for more information.

updatePartition: 0

disruptionBudget enables the creation of a PodDisruptionBudget to

prevent voluntary degrading of the Consul server cluster.

disruptionBudget: enabled: true

# maxUnavailable will default to (n/2)-1 where n is the number of
# replicas. If you'd like a custom value, you can specify an override here.
maxUnavailable: null

extraConfig is a raw string of extra configuration to set with the

server. This should be JSON.

extraConfig: | {}

extraVolumes is a list of extra volumes to mount. These will be exposed

to Consul in the path /consul/userconfig/<name>/. The value below is

an array of objects, examples are shown below.

extraVolumes: []

- type: secret (or "configMap")

#   name: my-secret
#   load: false # if true, will add to `-config-dir` to load by Consul

Affinity Settings

Commenting out or setting as empty the affinity variable, will allow

deployment to single node services such as Minikube

affinity: | podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution:

Client, when enabled, configures Consul clients to run on every node

within the Kube cluster. The current deployment model follows a traditional

DC where a single agent is deployed per node.

client: enabled: "-" image: null join: null

grpc should be set to true if the gRPC listener should be enabled.

This should be set to true if connectInject or meshGateway is enabled.

grpc: false

Resource requests, limits, etc. for the client cluster placement. This

should map directly to the value of the resources field for a PodSpec,

formatted as a multi-line string. By default no direct resource request

is made.

resources: null

extraConfig is a raw string of extra configuration to set with the

client. This should be JSON.

extraConfig: | {}

extraVolumes is a list of extra volumes to mount. These will be exposed

to Consul in the path /consul/userconfig/<name>/. The value below is

an array of objects, examples are shown below.

extraVolumes: []

- type: secret (or "configMap")

#   name: my-secret
#   load: false # if true, will add to `-config-dir` to load by Consul

Toleration Settings for Client pods

This should be a multi-line string matching the Toleration array

in a PodSpec.

The example below will allow Client pods to run on every node

regardless of taints

tolerations: |

- operator: "Exists"

tolerations: ""

nodeSelector labels for client pod assignment, formatted as a muli-line string.

ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector

Example:

nodeSelector: |

beta.kubernetes.io/arch: amd64

nodeSelector: null

Affinity Settings for Client pods, formatted as a multi-line YAML string.

ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity

Example:

affinity: |

nodeAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

nodeSelectorTerms:

- matchExpressions:

- key: node-role.kubernetes.io/master

operator: DoesNotExist

affinity: {}

used to assign priority to client pods

ref: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/

priorityClassName: ""

Extra annotations to attach to the client pods

This should be a multi-line string mapping directly to the a map of

the annotations to apply to the client pods

annotations: null

extraEnvVars is a list of extra enviroment variables to set with the pod. These could be

used to include proxy settings required for cloud auto-join feature,

in case kubernetes cluster is behind egress http proxies. Additionally, it could be used to configure

custom consul parameters.

extraEnvironmentVars: {}

http_proxy: http://localhost:3128,

# https_proxy: http://localhost:3128,
# no_proxy: internal.domain.com

snaphotAgent contains settings for setting up and running snapshot agents

within the Consul clusters. They are required to be co-located with Consul

clients, so will inherit the clients' nodeSelector, tolerations and affinity.

This is an Enterprise feature only.

snapshotAgent: enabled: false

# replicas determines how many snapshot agent pods are created
replicas: 2

# configSecret references a Kubernetes secret that should be manually created to
# contain the entire config to be used on the snapshot agent. This is the preferred
# method of configuration since there are usually storage credentials present.
# Snapshot agent config details:
# https://www.consul.io/docs/commands/snapshot/agent.html#config-file-options-
# To create a secret:
# https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret-using-kubectl-create-secret
configSecret:
  secretName: null
  secretKey: null

Configuration for DNS configuration within the Kubernetes cluster.

This creates a service that routes to all agents (client or server)

for serving DNS requests. This DOES NOT automatically configure kube-dns

today, so you must still manually configure a stubDomain with kube-dns

for this to have any effect:

https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#configure-stub-domain-and-upstream-dns-servers

dns: enabled: "-"

Set a predefined cluster IP for the DNS service.

Useful if you need to reference the DNS service's IP

address in CoreDNS config.

clusterIP: null

Extra annotations to attach to the dns service

This should be a multi-line string of

annotations to apply to the dns Service

annotations: null

ui:

True if you want to enable the Consul UI. The UI will run only

on the server nodes. This makes UI access via the service below (if

enabled) predictable rather than "any node" if you're running Consul

clients as well.

enabled: "-"

True if you want to create a Service entry for the Consul UI.

#

serviceType can be used to control the type of service created. For

example, setting this to "LoadBalancer" will create an external load

balancer (for supported K8S installations) to access the UI.

service: enabled: true type: null

This should be a multi-line string mapping directly to the a map of

# the annotations to apply to the UI service
annotations: null
# Additional ServiceSpec values
# This should be a multi-line string mapping directly to a Kubernetes
# ServiceSpec object.
additionalSpec: null

syncCatalog will run the catalog sync process to sync K8S with Consul

services. This can run bidirectional (default) or unidirectionally (Consul

to K8S or K8S to Consul only).

#

This process assumes that a Consul agent is available on the host IP.

This is done automatically if clients are enabled. If clients are not

enabled then set the node selection so that it chooses a node with a

Consul agent.

syncCatalog:

True if you want to enable the catalog sync. "-" for default.

enabled: true image: null default: false # true will sync by default, otherwise requires annotation

toConsul and toK8S control whether syncing is enabled to Consul or K8S

as a destination. If both of these are disabled, the sync will do nothing.

toConsul: true toK8S: true

k8sPrefix is the service prefix to prepend to services before registering

with Kubernetes. For example "consul-" will register all services

prepended with "consul-". (Consul -> Kubernetes sync)

k8sPrefix: null

k8sSourceNamespace is the Kubernetes namespace to watch for service

changes and sync to Consul. If this is not set then it will default

to all namespaces.

k8sSourceNamespace: null

consulPrefix is the service prefix which preprends itself

to Kubernetes services registered within Consul

For example, "k8s-" will register all services peprended with "k8s-".

(Kubernetes -> Consul sync)

consulPrefix: null

k8sTag is an optional tag that is applied to all of the Kubernetes services

that are synced into Consul. If nothing is set, defaults to "k8s".

(Kubernetes -> Consul sync)

k8sTag: null

syncClusterIPServices syncs services of the ClusterIP type, which may

or may not be broadly accessible depending on your Kubernetes cluster.

Set this to false to skip syncing ClusterIP services.

syncClusterIPServices: true

nodePortSyncType configures the type of syncing that happens for NodePort

services. The valid options are: ExternalOnly, InternalOnly, ExternalFirst.

- ExternalOnly will only use a node's ExternalIP address for the sync

- InternalOnly use's the node's InternalIP address

- ExternalFirst will preferentially use the node's ExternalIP address, but

if it doesn't exist, it will use the node's InternalIP address instead.

nodePortSyncType: ExternalFirst

aclSyncToken refers to a Kubernetes secret that you have created that contains

an ACL token for your Consul cluster which allows the sync process the correct

permissions. This is only needed if ACLs are enabled on the Consul cluster.

aclSyncToken: secretName: null secretKey: null

nodeSelector labels for syncCatalog pod assignment, formatted as a muli-line string.

ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector

Example:

nodeSelector: |

beta.kubernetes.io/arch: amd64

nodeSelector: null

Log verbosity level. One of "trace", "debug", "info", "warn", or "error".

logLevel: info

Override the default interval to perform syncing operations creating Consul services.

consulWriteInterval: null

ConnectInject will enable the automatic Connect sidecar injector.

connectInject: enabled: true image: null # image for consul-k8s that contains the injector default: false # true will inject by default, otherwise requires annotation

imageConsul and imageEnvoy can be set to Docker images for Consul and

Envoy, respectively. If the Consul image is not specified, the global

default will be used. If the Envoy image is not specified, an early

version of Envoy will be used.

imageConsul: null imageEnvoy: null

namespaceSelector is the selector for restricting the webhook to only

specific namespaces. This should be set to a multiline string.

namespaceSelector: null

The certs section configures how the webhook TLS certs are configured.

These are the TLS certs for the Kube apiserver communicating to the

webhook. By default, the injector will generate and manage its own certs,

but this requires the ability for the injector to update its own

MutatingWebhookConfiguration. In a production environment, custom certs

should probaly be used. Configure the values below to enable this.

certs:

secretName is the name of the secret that has the TLS certificate and

# private key to serve the injector webhook. If this is null, then the
# injector will default to its automatic management mode that will assign
# a service account to the injector to generate its own certificates.
secretName: null

# caBundle is a base64-encoded PEM-encoded certificate bundle for the
# CA that signed the TLS certificate that the webhook serves. This must
# be set if secretName is non-null.
caBundle: ""

# certName and keyName are the names of the files within the secret for
# the TLS cert and private key, respectively. These have reasonable
# defaults but can be customized if necessary.
certName: tls.crt
keyName: tls.key

nodeSelector labels for connectInject pod assignment, formatted as a muli-line string.

ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector

Example:

nodeSelector: |

beta.kubernetes.io/arch: amd64

nodeSelector: null

aclBindingRuleSelector accepts a query that defines which Service Accounts

can authenticate to Consul and receive an ACL token during Connect injection.

The default setting, i.e. serviceaccount.name!=default, prevents the

'default' Service Account from logging in.

If set to an empty string all service accounts can log in.

This only has effect if ACLs are enabled.

#

See https://www.consul.io/docs/acl/acl-auth-methods.html#binding-rules

and https://www.consul.io/docs/acl/auth-methods/kubernetes.html#trusted-identity-attributes

for more details.

Requires Consul >= v1.5 and consul-k8s >= v0.8.0.

aclBindingRuleSelector: "serviceaccount.name!=default"

Requires Consul >= v1.5 and consul-k8s >= v0.8.1.

centralConfig: enabled: false

# defaultProtocol allows you to specify a convenience default protocol if
# most of your services are of the same protocol type. The individual annotation
# on any given pod will override this value. A protocol must be provided,
# either through this setting or individual annotation, for a service to be
# registered correctly. Valid values are "http", "http2", "grpc" and "tcp".
defaultProtocol: null

# proxyDefaults is a raw json string that will be applied to all Connect
# proxy sidecar pods that can include any valid configuration for the
# configured proxy.
proxyDefaults: |
  {}

Mesh Gateways enable Consul Connect to work across Consul datacenters.

meshGateway:

If mesh gateways are enabled, a Deployment will be created that runs

gateways and Consul Connect will be configured to use gateways.

See https://www.consul.io/docs/connect/mesh_gateway.html

Requirements: consul >= 1.6.0 and consul-k8s >= 0.9.0 if using global.bootstrapACLs.

enabled: false

Globally configure which mode the gateway should run in.

Can be set to either "remote", "local", "none" or empty string or null.

See https://consul.io/docs/connect/mesh_gateway.html#modes-of-operation for

a description of each mode.

If set to anything other than "" or null, connectInject.centralConfig.enabled

should be set to true so that the global config will actually be used.

If set to the empty string, no global default will be set and the gateway mode

will need to be set individually for each service.

globalMode: local

Number of replicas for the Deployment.

replicas: 2

What gets registered as wan address for the gateway.

wanAddress:

Port that gets registered.

port: 443

# If true, each Gateway Pod will advertise its NodeIP
# (as provided by the Kubernetes downward API) as the wan address.
# This is useful if the node IPs are routable from other DCs.
# useNodeName and host must be false and "" respectively.
useNodeIP: true

# If true, each Gateway Pod will advertise its NodeName
# (as provided by the Kubernetes downward API) as the wan address.
# This is useful if the node names are DNS entries that are
# routable from other DCs.
# meshGateway.wanAddress.port will be used as the port for the wan address.
# useNodeIP and host must be false and "" respectively.
useNodeName: false

# If set, each gateway Pod will use this host as its wan address.
# Users must ensure that this address routes to the Gateway pods,
# for example via a DNS entry that routes to the Service fronting the Deployment.
# meshGateway.wanAddress.port will be used as the port for the wan address.
# useNodeIP and useNodeName must be false.
host: ""

The service option configures the Service that fronts the Gateway Deployment.

service:

Whether to create a Service or not.

enabled: false

# Type of service, ex. LoadBalancer, ClusterIP.
type: ClusterIP

# Port that the service will be exposed on.
# The targetPort will be set to meshGateway.containerPort.
port: 443

# Optional nodePort of the service. Can be used in conjunction with
# type: NodePort.
nodePort: null

# Optional YAML string for additional annotations.
annotations: null

# Optional YAML string that will be appended to the Service spec.
additionalSpec: null

Envoy image to use.

imageEnvoy: envoyproxy/envoy:v1.10.0

If set to true, gateway Pods will run on the host network.

hostNetwork: false

dnsPolicy to use.

dnsPolicy: null

Override the default 'mesh-gateway' service name registered in Consul.

Cannot be used if bootstrapACLs is true since the ACL token generated

is only for the name 'mesh-gateway'.

consulServiceName: ""

Port that the gateway will run on inside the container.

containerPort: 443

Optional hostPort for the gateway to be exposed on.

This can be used with wanAddress.port and wanAddress.useNodeIP

to expose the gateways directly from the node.

If hostNetwork is true, this must be null or set to the same port as

containerPort.

NOTE: Cannot set to 8500 or 8502 because those are reserved for the Consul

agent.

hostPort: null

If there are no connect-enabled services running, then the gateway

will fail health checks. You may disable health checks as a temporary

workaround.

enableHealthChecks: true

resources: | requests: memory: "128Mi" cpu: "250m" limits: memory: "256Mi" cpu: "500m"

By default, we set an anti affinity so that two gateway pods won't be

on the same node. NOTE: Gateways require that Consul client agents are

also running on the nodes alongside each gateway Pod.

affinity: | podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution:

Control whether a test Pod manifest is generated when running helm template.

When using helm install, the test Pod is not submitted to the cluster so this

is only useful when running helm template.

tests: enabled: true


I noticed that all components came up pretty quickly. I was also able to see my consul components on the consul ui. 
<img width="1306" alt="Screen Shot 2019-12-04 at 11 35 35 AM" src="https://user-images.githubusercontent.com/14130356/70162000-f7fbe580-168a-11ea-863b-585b12c4debc.png">
<img width="1015" alt="Screen Shot 2019-12-04 at 11 41 57 AM" src="https://user-images.githubusercontent.com/14130356/70162106-1d88ef00-168b-11ea-9ba6-9cbb260c09b8.png">

Notice the red check there? It is because this:

k logs static-server -c consul-connect-envoy-sidecar [2019-12-04 01:59:42.489][000001][info][main] [source/server/server.cc:206] initializing epoch 0 (hot restart version=10.200.16384.323.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=5931112) [2019-12-04 01:59:42.489][000001][info][main] [source/server/server.cc:208] statically linked extensions: [2019-12-04 01:59:42.489][000001][info][main] [source/server/server.cc:210] access_loggers: envoy.file_access_log,envoy.http_grpc_access_log [2019-12-04 01:59:42.489][000001][info][main] [source/server/server.cc:213] filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash [2019-12-04 01:59:42.489][000001][info][main] [source/server/server.cc:216] filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector [2019-12-04 01:59:42.489][000001][info][main] [source/server/server.cc:219] filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy [2019-12-04 01:59:42.489][000001][info][main] [source/server/server.cc:221] stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd [2019-12-04 01:59:42.489][000001][info][main] [source/server/server.cc:223] tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.zipkin [2019-12-04 01:59:42.489][000001][info][main] [source/server/server.cc:226] transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls [2019-12-04 01:59:42.489][000001][info][main] [source/server/server.cc:229] transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls [2019-12-04 01:59:42.501][000001][info][main] [source/server/server.cc:271] admin address: 127.0.0.1:19000 [2019-12-04 01:59:42.503][000001][info][config] [source/server/configuration_impl.cc:50] loading 0 static secret(s) [2019-12-04 01:59:42.503][000001][info][config] [source/server/configuration_impl.cc:56] loading 1 cluster(s) [2019-12-04 01:59:42.514][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:132] cm init: initializing cds [2019-12-04 01:59:42.517][000001][info][config] [source/server/configuration_impl.cc:67] loading 0 listener(s) [2019-12-04 01:59:42.517][000001][info][config] [source/server/configuration_impl.cc:92] loading tracing configuration [2019-12-04 01:59:42.517][000001][info][config] [source/server/configuration_impl.cc:112] loading stats sink configuration [2019-12-04 01:59:42.517][000001][info][main] [source/server/server.cc:463] starting main dispatch loop [2019-12-04 01:59:42.518][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 01:59:42.534][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 01:59:42.599][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 01:59:42.785][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 01:59:43.834][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 01:59:50.013][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:00:01.962][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:00:03.120][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:00:25.394][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:00:55.395][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:01:21.190][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:01:36.544][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:01:38.353][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:01:59.654][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:02:23.927][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:02:46.202][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:02:48.804][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:03:17.489][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:03:28.917][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:03:36.135][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers [2019-12-04 02:04:06.136][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers

So, right there we can see that even the base use case did not work. But I did not stop there, I went further with my experiment to see if I still get it working. I follow the documentation closely and created my server and client certs.

Set these values as show in this:

Available parameters and their default values for the Consul chart.

global:

enabled is the master enabled switch. Setting this to true or false

will enable or disable all the components within this chart by default.

Each component can be overridden using the component-specific "enabled"

value.

enabled: true

Domain to register the Consul DNS server to listen for.

domain: consul

Image is the name (and tag) of the Consul Docker image for clients and

servers below. This can be overridden per component.

#

Examples:

image: "consul:1.5.0"

image: "hashicorp/consul-enterprise:1.5.0-ent" # Enterprise Consul image

image: "consul:1.6.2"

imageK8S is the name (and tag) of the consul-k8s Docker image that

is used for functionality such as the catalog sync. This can be overridden

per component below.

Note: support for the catalog sync's liveness and readiness probes was added

to consul-k8s v0.6.0. If using an older consul-k8s version, you may need to

remove these checks to make the sync work.

If using mesh gateways and bootstrapACLs then must be >= 0.9.0.

imageK8S: "hashicorp/consul-k8s:0.9.4"

Datacenter is the name of the datacenter that the agents should register

as. This shouldn't be changed once the Consul cluster is up and running

since Consul doesn't support an automatic way to change this value

currently: https://github.com/hashicorp/consul/issues/1858

datacenter: dc1

enablePodSecurityPolicies is a boolean flag that controls whether pod

security policies are created for the consul components created by this

chart. See https://kubernetes.io/docs/concepts/policy/pod-security-policy/

enablePodSecurityPolicies: false

Gossip encryption key. To enable gossip encryption, provide the name of

a Kubernetes secret that contains a gossip key. You can create a gossip

key with the "consul keygen" command.

See https://www.consul.io/docs/commands/keygen.html

gossipEncryption: secretName: ${consul_gossip_secret_name} secretKey: ${consul_gossip_secret_key}

bootstrapACLs will automatically create and assign ACL tokens within

the Consul cluster. This currently requires enabling both servers and

clients within Kubernetes. Additionally requires Consul v1.4+ and

consul-k8s v0.8.0+.

bootstrapACLs: true

Server, when enabled, configures a server cluster to run. This should

be disabled if you plan on connecting to a Consul cluster external to

the Kube cluster.

server: enabled: "-" image: null replicas: 3 bootstrapExpect: 3 # Should <= replicas count

enterpriseLicense refers to a Kubernetes secret that you have created that

contains your enterprise license. It is required if you are using an

enterprise binary. Defining it here applies it to your cluster once a leader

has been elected. If you are not using an enterprise image

or if you plan to introduce the license key via another route, then set

these fields to null.

enterpriseLicense: secretName: null secretKey: null

storage and storageClass are the settings for configuring stateful

storage for the server pods. storage should be set to the disk size of

the attached volume. storageClass is the class of storage which defaults

to null (the Kube cluster will pick the default).

storage: ${consul_volume_size} storageClass: null

connect will enable Connect on all the servers, initializing a CA

for Connect-related connections. Other customizations can be done

via the extraConfig setting.

connect: true

Resource requests, limits, etc. for the server cluster placement. This

should map directly to the value of the resources field for a PodSpec,

formatted as a multi-line string. By default no direct resource request

is made.

resources: null

updatePartition is used to control a careful rolling update of Consul

servers. This should be done particularly when changing the version

of Consul. Please refer to the documentation for more information.

updatePartition: 0

disruptionBudget enables the creation of a PodDisruptionBudget to

prevent voluntary degrading of the Consul server cluster.

disruptionBudget: enabled: false

# maxUnavailable will default to (n/2)-1 where n is the number of
# replicas. If you'd like a custom value, you can specify an override here.
maxUnavailable: null

extraConfig is a raw string of extra configuration to set with the

server. This should be JSON.

extraConfig: | { "ui": true, "log_level": "info", "client_addr": "0.0.0.0", "enable_script_checks": false, "disable_remote_exec": true, "verify_incoming": false, "verify_outgoing": true, "verify_server_hostname": true, "ca_file": "/consul/userconfig/${consul_server_tls_secret}/ca.pem", "cert_file": "/consul/userconfig/${consul_server_tls_secret}/tls.crt", "key_file": "/consul/userconfig/${consul_server_tls_secret}/tls.key", "ports": { "http": -1, "https": 8500 } }

extraVolumes is a list of extra volumes to mount. These will be exposed

to Consul in the path /consul/userconfig/<name>/. The value below is

an array of objects, examples are shown below.

extraVolumes:

Client, when enabled, configures Consul clients to run on every node

within the Kube cluster. The current deployment model follows a traditional

DC where a single agent is deployed per node.

client: enabled: "-" image: null join: null

grpc should be set to true if the gRPC listener should be enabled.

This should be set to true if connectInject or meshGateway is enabled.

grpc: true

Resource requests, limits, etc. for the client cluster placement. This

should map directly to the value of the resources field for a PodSpec,

formatted as a multi-line string. By default no direct resource request

is made.

resources: null

extraConfig is a raw string of extra configuration to set with the

client. This should be JSON.

extraConfig: | { "log_level": "info", "verify_incoming": true, "verify_outgoing": true, "verify_server_hostname": true, "ca_file": "/consul/userconfig/${consul_client_tls_secret}/ca.pem", "cert_file": "/consul/userconfig/${consul_client_tls_secret}/tls.crt", "key_file": "/consul/userconfig/${consul_client_tls_secret}/tls.key",

  "ports": {
    "http": -1,
    "https": 8500
  }
}

extraVolumes is a list of extra volumes to mount. These will be exposed

to Consul in the path /consul/userconfig/<name>/. The value below is

an array of objects, examples are shown below.

extraVolumes:

Configuration for DNS configuration within the Kubernetes cluster.

This creates a service that routes to all agents (client or server)

for serving DNS requests. This DOES NOT automatically configure kube-dns

today, so you must still manually configure a stubDomain with kube-dns

for this to have any effect:

https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#configure-stub-domain-and-upstream-dns-servers

dns: enabled: "-"

Set a predefined cluster IP for the DNS service.

Useful if you need to reference the DNS service's IP

address in CoreDNS config.

clusterIP: null

Extra annotations to attach to the dns service

This should be a multi-line string of

annotations to apply to the dns Service

annotations: null

ui:

True if you want to enable the Consul UI. The UI will run only

on the server nodes. This makes UI access via the service below (if

enabled) predictable rather than "any node" if you're running Consul

clients as well.

enabled: "-"

True if you want to create a Service entry for the Consul UI.

#

serviceType can be used to control the type of service created. For

example, setting this to "LoadBalancer" will create an external load

balancer (for supported K8S installations) to access the UI.

service: enabled: true type: ${consul_service_type}

This should be a multi-line string mapping directly to the a map of

# the annotations to apply to the UI service
annotations: null
# Additional ServiceSpec values
# This should be a multi-line string mapping directly to a Kubernetes
# ServiceSpec object.
additionalSpec: null

syncCatalog will run the catalog sync process to sync K8S with Consul

services. This can run bidirectional (default) or unidirectionally (Consul

to K8S or K8S to Consul only).

#

This process assumes that a Consul agent is available on the host IP.

This is done automatically if clients are enabled. If clients are not

enabled then set the node selection so that it chooses a node with a

Consul agent.

syncCatalog:

True if you want to enable the catalog sync. "-" for default.

enabled: true image: null default: false # true will sync by default, otherwise requires annotation

toConsul and toK8S control whether syncing is enabled to Consul or K8S

as a destination. If both of these are disabled, the sync will do nothing.

toConsul: true toK8S: false

k8sPrefix is the service prefix to prepend to services before registering

with Kubernetes. For example "consul-" will register all services

prepended with "consul-". (Consul -> Kubernetes sync)

k8sPrefix: null

k8sSourceNamespace is the Kubernetes namespace to watch for service

changes and sync to Consul. If this is not set then it will default

to all namespaces.

k8sSourceNamespace: null

consulPrefix is the service prefix which preprends itself

to Kubernetes services registered within Consul

For example, "k8s-" will register all services peprended with "k8s-".

(Kubernetes -> Consul sync)

consulPrefix: null

k8sTag is an optional tag that is applied to all of the Kubernetes services

that are synced into Consul. If nothing is set, defaults to "k8s".

(Kubernetes -> Consul sync)

k8sTag: null

syncClusterIPServices syncs services of the ClusterIP type, which may

or may not be broadly accessible depending on your Kubernetes cluster.

Set this to false to skip syncing ClusterIP services.

syncClusterIPServices: true

nodePortSyncType configures the type of syncing that happens for NodePort

services. The valid options are: ExternalOnly, InternalOnly, ExternalFirst.

- ExternalOnly will only use a node's ExternalIP address for the sync

- InternalOnly use's the node's InternalIP address

- ExternalFirst will preferentially use the node's ExternalIP address, but

if it doesn't exist, it will use the node's InternalIP address instead.

nodePortSyncType: ExternalFirst

aclSyncToken refers to a Kubernetes secret that you have created that contains

an ACL token for your Consul cluster which allows the sync process the correct

permissions. This is only needed if ACLs are enabled on the Consul cluster.

aclSyncToken: secretName: ${consul_acl_sync_secret} secretKey: ${consul_acl_sync_key}

nodeSelector labels for syncCatalog pod assignment, formatted as a muli-line string.

ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector

Example:

nodeSelector: |

beta.kubernetes.io/arch: amd64

nodeSelector: null

Log verbosity level. One of "trace", "debug", "info", "warn", or "error".

logLevel: trace

Override the default interval to perform syncing operations creating Consul services.

consulWriteInterval: null

ConnectInject will enable the automatic Connect sidecar injector.

connectInject: enabled: true image: null # image for consul-k8s that contains the injector default: true # true will inject by default, otherwise requires annotation

imageConsul and imageEnvoy can be set to Docker images for Consul and

Envoy, respectively. If the Consul image is not specified, the global

default will be used. If the Envoy image is not specified, an early

version of Envoy will be used.

imageConsul: null imageEnvoy: null

namespaceSelector is the selector for restricting the webhook to only

specific namespaces. This should be set to a multiline string.

namespaceSelector: null

The certs section configures how the webhook TLS certs are configured.

These are the TLS certs for the Kube apiserver communicating to the

webhook. By default, the injector will generate and manage its own certs,

but this requires the ability for the injector to update its own

MutatingWebhookConfiguration. In a production environment, custom certs

should probaly be used. Configure the values below to enable this.

certs:

secretName is the name of the secret that has the TLS certificate and

# private key to serve the injector webhook. If this is null, then the
# injector will default to its automatic management mode that will assign
# a service account to the injector to generate its own certificates.
secretName: ${consul_client_tls_secret}

# caBundle is a base64-encoded PEM-encoded certificate bundle for the
# CA that signed the TLS certificate that the webhook serves. This must
# be set if secretName is non-null.
caBundle: ${tls_ca_file}

# certName and keyName are the names of the files within the secret for
# the TLS cert and private key, respectively. These have reasonable
# defaults but can be customized if necessary.
certName: "tls.crt"
keyName: "tls.key"

nodeSelector labels for connectInject pod assignment, formatted as a multi-line string.

ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector

Example:

nodeSelector: |

beta.kubernetes.io/arch: amd64

nodeSelector: null

aclBindingRuleSelector accepts a query that defines which Service Accounts

can authenticate to Consul and receive an ACL token during Connect injection.

The default setting, i.e. serviceaccount.name!=default, prevents the

'default' Service Account from logging in.

If set to an empty string all service accounts can log in.

This only has effect if ACLs are enabled.

#

See https://www.consul.io/docs/acl/acl-auth-methods.html#binding-rules

and https://www.consul.io/docs/acl/auth-methods/kubernetes.html#trusted-identity-attributes

for more details.

Requires Consul >= v1.5 and consul-k8s >= v0.8.0.

aclBindingRuleSelector: "serviceaccount.name!=default"

Requires Consul >= v1.5 and consul-k8s >= v0.8.1.

centralConfig: enabled: false

# defaultProtocol allows you to specify a convenience default protocol if
# most of your services are of the same protocol type. The individual annotation
# on any given pod will override this value. A protocol must be provided,
# either through this setting or individual annotation, for a service to be
# registered correctly. Valid values are "http", "http2", "grpc" and "tcp".
defaultProtocol: null

# proxyDefaults is a raw json string that will be applied to all Connect
# proxy sidecar pods that can include any valid configuration for the
# configured proxy.
proxyDefaults: |
  {}

Mesh Gateways enable Consul Connect to work across Consul datacenters.

meshGateway:

If mesh gateways are enabled, a Deployment will be created that runs

gateways and Consul Connect will be configured to use gateways.

See https://www.consul.io/docs/connect/mesh_gateway.html

Requirements: consul >= 1.6.0 and consul-k8s >= 0.9.0 if using global.bootstrapACLs.

enabled: false

Globally configure which mode the gateway should run in.

Can be set to either "remote", "local", "none" or empty string or null.

See https://consul.io/docs/connect/mesh_gateway.html#modes-of-operation for

a description of each mode.

If set to anything other than "" or null, connectInject.centralConfig.enabled

should be set to true so that the global config will actually be used.

If set to the empty string, no global default will be set and the gateway mode

will need to be set individually for each service.

globalMode: local

Number of replicas for the Deployment.

replicas: 2

What gets registered as wan address for the gateway.

wanAddress:

Port that gets registered.

port: 443

# If true, each Gateway Pod will advertise its NodeIP
# (as provided by the Kubernetes downward API) as the wan address.
# This is useful if the node IPs are routable from other DCs.
# useNodeName and host must be false and "" respectively.
useNodeIP: true

# If true, each Gateway Pod will advertise its NodeName
# (as provided by the Kubernetes downward API) as the wan address.
# This is useful if the node names are DNS entries that are
# routable from other DCs.
# meshGateway.wanAddress.port will be used as the port for the wan address.
# useNodeIP and host must be false and "" respectively.
useNodeName: false

# If set, each gateway Pod will use this host as its wan address.
# Users must ensure that this address routes to the Gateway pods,
# for example via a DNS entry that routes to the Service fronting the Deployment.
# meshGateway.wanAddress.port will be used as the port for the wan address.
# useNodeIP and useNodeName must be false.
host: ""

The service option configures the Service that fronts the Gateway Deployment.

service:

Whether to create a Service or not.

enabled: false

# Type of service, ex. LoadBalancer, ClusterIP.
type: ClusterIP

# Port that the service will be exposed on.
# The targetPort will be set to meshGateway.containerPort.
port: 443

# Optional nodePort of the service. Can be used in conjunction with
# type: NodePort.
nodePort: null

# Optional YAML string for additional annotations.
annotations: null

# Optional YAML string that will be appended to the Service spec.
additionalSpec: null

Envoy image to use.

imageEnvoy: envoyproxy/envoy:v1.10.0

If set to true, gateway Pods will run on the host network.

hostNetwork: false

dnsPolicy to use.

dnsPolicy: null

Override the default 'mesh-gateway' service name registered in Consul.

Cannot be used if bootstrapACLs is true since the ACL token generated

is only for the name 'mesh-gateway'.

consulServiceName: ""

Port that the gateway will run on inside the container.

containerPort: 443

Optional hostPort for the gateway to be exposed on.

This can be used with wanAddress.port and wanAddress.useNodeIP

to expose the gateways directly from the node.

If hostNetwork is true, this must be null or set to the same port as

containerPort.

NOTE: Cannot set to 8500 or 8502 because those are reserved for the Consul

agent.

hostPort: null

If there are no connect-enabled services running, then the gateway

will fail health checks. You may disable health checks as a temporary

workaround.

enableHealthChecks: true

resources: | requests: memory: "128Mi" cpu: "250m" limits: memory: "256Mi" cpu: "500m"

By default, we set an anti affinity so that two gateway pods won't be

on the same node. NOTE: Gateways require that Consul client agents are

also running on the nodes alongside each gateway Pod.

affinity: | podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution:

Control whether a test Pod manifest is generated when running helm template.

When using helm install, the test Pod is not submitted to the cluster so this

is only useful when running helm template.

tests: enabled: true


Even with that I ran into some issues and was never able to get everything working. I need someone to tell me where I went wrong here. If I can get it working I will create a readme file for this chart. Oh, I also had to change liveliness and readiness  probes to this:

livenessProbe: exec: command:

ishustava commented 4 years ago

Hey @bsakweson,

Thanks for creating this issue and sorry you had a hard time installing the chart.

Your first issue is because Connect requires client.grpc to be set to true. This is documented here, but it's definitely been a common source of misconfiguration issues. In the next release we're going to change that to be enabled by default with #282, so hopefully, it will help with that in the future.

Your second issue is because Connect injection doesn't properly support clients with TLS configured (hashicorp/consul-k8s#79 and hashicorp/consul-k8s#154). This something we're actively working on right now and hope to ship a fix for this soon!

For which resources did you have to change the liveness and readiness probes?

bsakweson commented 4 years ago

@ishustava, please take a look at this #74 that is where I got that from. Thanks a bunch for your quick response.

lkysow commented 4 years ago

Hi @bsakweson, we're working on full TLS support as we speak and will have a release in January. Please follow this ticket: https://github.com/hashicorp/consul-helm/issues/136 for updates.

I'm going to close this because our focus is on getting that release out and that should close this issue.

bsakweson commented 4 years ago

Do you mind sharing an alternate way if there is any? Closing this issue without any resolution is not helpful for anyone who may run into the same issue. Wouldn't you agree?

lkysow commented 4 years ago

You're trying to get TLS working yourself right? There isn't an alternative way that I know of except to wait for us to finish the feature. Someone else running into the same issue is advised to wait on the main feature ticket tracking TLS support.

Sorry for closing the ticket without resolution but I was treating it as a duplicate of the TLS support ticket. We don't want to have duplicate issues. But maybe I'm mistaken, is this different?

On Fri, Dec 20, 2019, 4:51 PM bsakweson notifications@github.com wrote:

Do you mind sharing an alternate way if there is any? Closing this issue without any resolution is not helpful for anyone who may run into the same issue. Wouldn't you agree?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/hashicorp/consul-helm/issues/300?email_source=notifications&email_token=AAH4RPPWV27T6NLSKDMOVT3QZVSAPA5CNFSM4JVLYIF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHORD6A#issuecomment-568136184, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH4RPIWJWDHODJYP7JDCV3QZVSAPANCNFSM4JVLYIFQ .

ishustava commented 4 years ago

Hey @bsakweson, sorry for this delayed response. I just wanted to let you know that we have released TLS support in Helm release v0.16.x. You shouldn't need to use extraConfig anymore and all consul-k8s components, like sync catalog and Connect properly support it. I hope you could give it a try and it will work around the issues you've experienced previously.

bsakweson commented 4 years ago

Awesome! Thanks for the heads-up, I will check it out this weekend.

On Thu, Mar 19, 2020, 17:43 Iryna Shustava notifications@github.com wrote:

Hey @bsakweson https://github.com/bsakweson, sorry for this delayed response. I just wanted to let you know that we have released TLS support in Helm release v0.16.x. You shouldn't need to use extraConfig anymore and all consul-k8s components, like sync catalog and Connect properly support it. I hope you could give it a try and it will work around the issues you've experienced previously.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hashicorp/consul-helm/issues/300#issuecomment-601431014, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLZZNBXYN6JX56XIW457O3RIKGYDANCNFSM4JVLYIFQ .