Envoy fails to start when Route references non-existent Cluster

[nulltrope]

Title: Envoy fails to start when Route references non-existent Cluster

Description:

Envoy fails to initialize when attempting to start with a Route which references a non-existent Cluster. This came as a surprise to me, since http filters such as the http.ext_authz can reference a non-existent Cluster without issue.

I found a config option config.route.v3.RouteAction.ClusterNotFoundResponseCode in the docs, however this seems to only apply when using cluster_header to route and not cluster explicitly.

In environments using CDS for Cluster discovery but still statically defining routes, I can see value in being able to start Envoy with a cluster which does not yet exist, but will soon be added dynamically.

Repro steps:

Run Envoy with the configuration below, it will fail to start.

Admin and Stats Output:

N/A, Envoy failing to start

Config:

admin:
  access_log_path: "/dev/stdout"
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901}
dynamic_resources:
  ads_config:
    api_type: GRPC
    transport_api_version: V3
    grpc_services:
      envoy_grpc:
        cluster_name: xds_cluster
  cds_config:
    resource_api_version: V3
    ads: {}
static_resources:
  listeners:
  - name: ingress
    address:
      socket_address: { address: 0.0.0.0, port_value: 9411 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress
          codec_type: AUTO
          route_config:
            name: ingress_route
            virtual_hosts: 
            - name: ingress_virtual_host
              domains: 
              - "*"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: non-existent-cluster
                  cluster_not_found_response_code: NOT_FOUND
          http_filters:
          - name: envoy.filters.http.ext_authz
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
              stat_prefix: ext_authz
              transport_api_version: V3
              grpc_service:
                envoy_grpc:
                  cluster_name: non-existent-cluster
          - name: envoy.router
          access_log:
          - name: envoy.file_access_log
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: "/dev/stdout"
              format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH):256% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n"
  clusters:
  - name: xds_cluster
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    http2_protocol_options: {}
    load_assignment:
      cluster_name: xds_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: discovery
                port_value: 10000

Logs:

[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:323] initializing epoch 0 (base id=0, hot restart version=11.104)
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:325] statically linked extensions:
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.quic_client_codec: quiche
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.stats_sinks: envoy.dog_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.compression.decompressor: envoy.compression.gzip.decompressor
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.thrift_proxy.filters: envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.http.cache: envoy.extensions.http.cache.simple
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.bootstrap: envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.gzip, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.squash, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.compression.compressor: envoy.compression.gzip.compressor
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.resolvers: envoy.ip
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.quic_server_codec: quiche
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.udp_packet_writers: udp_default_writer, udp_gso_batch_writer
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.postgres_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.rocketmq_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.dubbo_proxy.route_matchers: default
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.udp_listeners: quiche_quic_listener, raw_udp_listener
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.upstreams.http.http_protocol_options
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, tls
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.health_checkers: envoy.health_checkers.redis
[2021-06-07 15:51:13.876][1][info][main] [source/server/server.cc:327]   envoy.dubbo_proxy.protocols: dubbo
[2021-06-07 15:51:13.890][1][info][main] [source/server/server.cc:343] HTTP header map info:
[2021-06-07 15:51:13.891][1][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2021-06-07 15:51:13.891][1][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2021-06-07 15:51:13.892][1][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2021-06-07 15:51:13.892][1][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2021-06-07 15:51:13.892][1][info][main] [source/server/server.cc:346]   request header map: 616 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-method,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-proto,x-ot-span-context,x-request-id
[2021-06-07 15:51:13.892][1][info][main] [source/server/server.cc:346]   request trailer map: 128 bytes:
[2021-06-07 15:51:13.892][1][info][main] [source/server/server.cc:346]   response header map: 424 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2021-06-07 15:51:13.892][1][info][main] [source/server/server.cc:346]   response trailer map: 152 bytes: grpc-message,grpc-status
[2021-06-07 15:51:13.893][1][info][main] [source/server/server.cc:486] admin address: 0.0.0.0:9901
[2021-06-07 15:51:13.894][1][info][main] [source/server/server.cc:634] runtime: layers:
  - name: base
    static_layer:
      {}
  - name: admin
    admin_layer:
      {}
[2021-06-07 15:51:13.895][1][info][config] [source/server/configuration_impl.cc:125] loading tracing configuration
[2021-06-07 15:51:13.895][1][info][config] [source/server/configuration_impl.cc:85] loading 0 static secret(s)
[2021-06-07 15:51:13.895][1][info][config] [source/server/configuration_impl.cc:91] loading 1 cluster(s)
[2021-06-07 15:51:13.896][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 14, Cluster not available
[2021-06-07 15:51:13.897][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:63] Unable to establish new stream
[2021-06-07 15:51:13.897][1][info][config] [source/server/configuration_impl.cc:95] loading 1 listener(s)
[2021-06-07 15:51:13.905][1][critical][main] [source/server/server.cc:109] error initializing configuration '/etc/envoy-config.yaml': route: unknown cluster 'non-existent-cluster'
[2021-06-07 15:51:13.906][1][info][main] [source/server/server.cc:782] exiting
route: unknown cluster 'non-existent-cluster'

Call Stack:

N/A

[wbpcode]

It is a design, not a bug. Envoy will verify the existence of the clusters in the static route table. If really necessary, you can submit a feature request to let envoy provide an option to turn off this verification.

[wbpcode]

In addition, there is a possible solution, of course I haven't tried it.

That is to replace your static route table in HCM with rds based on local files. You only need to split your configuration file into two: one is the main configuration file and the other is an rds file.

[zuercher]

This is by design. For static routes with CDS, see the validate_clusters field on the RouteConfiguration.

[nulltrope]

Thanks @zuercher! That option is exactly what I was looking for.