Configure MaxFailures on Ingress Gateway

Joxit commented 2 years ago

Question

Hi there.

TLDR; How to configure MaxFailures or envoy consecutive_5xx of Ingress Gateway ?

I'm using consul for years and now I want to use it with Kubernetes. I'm working with services inside and outside Kubernetes, so I must use the Ingress Gateway for the communication. The issue with Ingress Gateway (and envoy dynamic configuration) is the default consecutive 5xx configuration. This is irrelevant in our use and degrades our performance when returning 5xx. After 5 consecutive 5xx from a backend, envoy returns this message: no healthy upstream.

I check the consul main repository and found this issue https://github.com/hashicorp/consul/issues/11422 but this did not work (maybe because we should use ProxyDefaults for Ingress Gateway?).

I also checked the documentation for ProxyDefaults Kubernetes YAML with the link to envoy proxy config options and saw this key passive_health_check but did not understand if I can and where I can add this in the proxy defaults configuration...

In my example bellow, the service returns the wanted status code curl http://127.0.0.1:8080/status?code=500 will return a status 500 code.

CLI Commands (consul-k8s, consul-k8s-control-plane, helm)

helm upgrade --install --version 0.39.0 --create-namespace -n consul --set global.name=consul -f values.yaml consul hashicorp/consul

Helm Configuration

Steps to reproduce this issue, eg:

When running helm install with the following values.yaml:


global:
datacenter: 'dc1'
tls:
enabled: false
acls:
manageSystemACLs: false
image: 'hashicorp/consul:1.11.1'
imageEnvoy: 'envoyproxy/envoy-alpine:v1.20.0'
imageK8S: 'hashicorp/consul-k8s-control-plane:0.39.0'
metrics:
enabled: false
meshGateway:
enabled: false

ingressGateways: enabled: true defaults: replicas: 1 service: ports:

port: 30000 nodePort: 30000 gateways:
- name: ingress-gateway service: type: NodePort terminatingGateways: enabled: false connectInject: enabled: true default: true replicas: 1
metrics: defaultEnabled: false server: enabled: true replicas: 1 externalServers: enabled: false client: grpc: true extraConfig: | {"advertise_reconnect_timeout": "1h"} controller: enabled: true dns: enabled: true syncCatalog: enabled: false
```
2. The `ingress-gateway.yaml` manifest
```yml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
spec:
config:
protocol: 'http'
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: IngressGateway
metadata:
name: ingress-gateway
spec:
listeners:
- port: 30000
  protocol: http
  services:
```

name: statuscode


3. The `status-code.yaml` manifest
```yaml
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: statuscode
name: statuscode
spec:
ports:
- port: 8080
selector:
app.kubernetes.io/name: statuscode
status:
loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: statuscode
name: statuscode
spec:
replicas: 1
selector:
matchLabels:
  app.kubernetes.io/name: statuscode
strategy: {}
template:
metadata:
  labels:
    app.kubernetes.io/name: statuscode
  annotations:
    'consul.hashicorp.com/connect-inject': 'true'
    'consul.hashicorp.com/connect-service': 'statuscode'
    'consul.hashicorp.com/connect-port': '8080'
spec:
  containers:

image: dotnsf/statuscode_generator name: statuscode ports:
- containerPort: 8080 resources: requests: { memory: '256Mi' } limits: { memory: '1Gi' } readinessProbe: httpGet: { path: /, port: 8080 } initialDelaySeconds: 20 periodSeconds: 10 livenessProbe: httpGet: { path: /, port: 8080 } initialDelaySeconds: 20 periodSeconds: 10 restartPolicy: Always status: {}
  
  apiVersion: consul.hashicorp.com/v1alpha1 kind: ServiceDefaults metadata: name: statuscode spec: protocol: http upstreamConfig: defaults: passiveHealthCheck: maxFailures: 400 interval: '1s'
  
  apiVersion: consul.hashicorp.com/v1alpha1 kind: ServiceIntentions metadata: name: statuscode spec: destination: name: statuscode sources:
  - name: 'ingress-gateway' action: allow

Logs

When I check the ingress-gateway config dump (http://127.0.0.1:19000/config_dump), where is what I found, outlier_detection still empty:

kubectl port-forward -n consul consul-ingress-gateway-5d9dc99bbd-shw5r 19000
curl http://127.0.0.1:19000/config_dump

     {
          "version_info": "ea72e0ee8944a6e8a3d74828cd48c542dfbb8c70b59be52e83d4d0159f18e4af",
          "cluster": {
            "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
            "name": "statuscode.default.dc1.internal.0ad86cd7-4cfa-e946-8f67-b920776a6757.consul",
            "type": "EDS",
            "eds_cluster_config": {
              "eds_config": {
                "ads": {},
                "resource_api_version": "V3"
              }
            },
            "connect_timeout": "5s",
            "circuit_breakers": {},
            "outlier_detection": {},

You can access the Ingress Gateway on your computer on port 30000, send 6 times this request:

curl -H 'Host: statuscode.ingress.consul' http://<ingress-gateway-ip>:30000/status?code=500

Current understanding and Expected behavior

Environment details

consul-k8s version: 0.39.0
values.yaml used to deploy the helm chart: :arrow_up:

Additionally, please provide details regarding the Kubernetes Infrastructure, as shown below:

Kubernetes version: minikube v1.23.2

Additional Context

david-yu commented 2 years ago

Hi @Joxit we are aware that the connection limits are not configurable on ingress gateway's cluster config - max_connections, max_pending_requests, and max_requests are all at the default (1024). I'm going to transfer this to Core as we'd need to expose via a Config Entry field.

Amier3 commented 2 years ago

Hey @Joxit

Just an update that we didn't forget about this. This has come up a few times and we're currently tracking this in https://github.com/hashicorp/consul/issues/8722 . We use 👍 's as a way to try and gauge the popularity of a feature request, so I encourage you to thumbs up that issue.

Joxit commented 2 years ago

Hi, thanks for the update 😊

hashicorp / consul