Gloo Edge doesn't support a hybrid of gateway and knative profiles

Describe the bug Please see this comment: https://github.com/solo-io/gloo/issues/1544#issuecomment-547669320

I want to use both the Gateway (Virtual Services) and Knative together to route requests in a similar manner as described in https://github.com/solo-io/gloo/issues/1544

It is stated in the comment that you can "Simply install Gloo in gateway mode on top of the existing knative install (or vice versa): glooctl install gateway."

However, glooctl simple doesn't let you do that:

To Reproduce Steps to reproduce the behavior:

glooctl install knative
glooctl install gateway
Error: installing gloo edge in gateway mode: Gloo has already been installed to namespace gloo-system

Expected behavior Installing the gateway is successful.

Additional context glooctl version Client: {"version":"1.5.14"}

I went through the Helm install to enable both Knative and Gateway, and see if that was possible rather than using glooctl. If I set knative.proxy.service.type=ClusterIP everything seems to deploy OK. With type=LoadBalancer the knative-proxy will not deploy.

(install Knative without Gloo)
glooctl install knative -g
kubectl create namespace gloo-system
helm install gloo gloo/gloo --namespace gloo-system \
  --set gateway.enabled=true,settings.integrations.knative.enabled=true,settings.integrations.knative.version=v0.10.0,settings.integrations.knative.proxy.service.type=ClusterIP
kubectl get all -n gloo-system

Result:

NAME                                          READY   STATUS    RESTARTS   AGE
pod/knative-internal-proxy-7cd8448b58-sm5gq   1/1     Running   0          35s
pod/knative-external-proxy-f4f67644d-5wpxt    1/1     Running   0          35s
pod/gateway-proxy-78669994ff-6rnvq            1/1     Running   0          35s
pod/svclb-gateway-proxy-z5d9q                 2/2     Running   0          35s
pod/ingress-6f7f8bdcd9-lk5ft                  1/1     Running   1          35s
pod/gloo-6d86dc87c8-rqx7r                     1/1     Running   1          35s
pod/discovery-5dc96cbb4d-dmjkl                1/1     Running   1          35s
pod/gateway-7fbfc6cb8f-w9lht                  1/1     Running   1          35s

NAME                             TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                               AGE
service/gloo                     ClusterIP      10.43.224.75    <none>        9977/TCP,9976/TCP,9988/TCP,9979/TCP   35s
service/knative-external-proxy   ClusterIP      10.43.41.29     <none>        80/TCP,443/TCP                        35s
service/knative-internal-proxy   ClusterIP      10.43.120.17    <none>        80/TCP,443/TCP                        35s
service/gateway                  ClusterIP      10.43.191.186   <none>        443/TCP                               35s
service/gateway-proxy            LoadBalancer   10.43.220.143   172.19.0.2    80:31162/TCP,443:30813/TCP            35s

NAME                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/svclb-gateway-proxy   1         1         1       1            1           <none>          35s

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/knative-internal-proxy   1/1     1            1           35s
deployment.apps/knative-external-proxy   1/1     1            1           35s
deployment.apps/gateway-proxy            1/1     1            1           35s
deployment.apps/ingress                  1/1     1            1           35s
deployment.apps/gloo                     1/1     1            1           35s
deployment.apps/discovery                1/1     1            1           35s
deployment.apps/gateway                  1/1     1            1           35s

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/knative-internal-proxy-7cd8448b58   1         1         1       35s
replicaset.apps/knative-external-proxy-f4f67644d    1         1         1       35s
replicaset.apps/gateway-proxy-78669994ff            1         1         1       35s
replicaset.apps/ingress-6f7f8bdcd9                  1         1         1       35s
replicaset.apps/gloo-6d86dc87c8                     1         1         1       35s
replicaset.apps/discovery-5dc96cbb4d                1         1         1       35s
replicaset.apps/gateway-7fbfc6cb8f                  1         1         1       35s

However, there is a problem with the Knative service deployment (IngressNotConfigured):

kubectl apply --filename petstore.yaml
service.serving.knative.dev/petstore created 
kubectl get kservice                                                                                         
NAME       URL                                   LATESTCREATED    LATESTREADY      READY     REASON
petstore   http://petstore.default.example.com   petstore-2bqpg   petstore-2bqpg   Unknown   IngressNotConfigured

Just to provide some clarity around the use case, as documented in the referenced issue above, I'd be looking to do something like this:

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: 'petstore'
  namespace: 'gloo-system'
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
       - prefix: '/all-pets'
      routeAction:
        single:
          upstream:
            name: 'gloo-system-knative-internal-proxy-80'
            namespace: 'gloo-system'
      options:
        prefixRewrite: '/api/pets'
        hostRewrite: 'petstore.default.example.com'
status: {}

Hi @murphye . glooctl install knative and glooctl install gateway are meant to be default installations for simpler use cases. You can install both at the same time using glooctl or helm by providing the proper helm value overrides. For example, the following values.yaml:

gateway:
  enabled: true
settings:
  integrations:
    knative:
      enabled: false

and then glooctl install gateway -f values.yaml or helm install --values values.yaml

Hi @kdorosh Please check my above comment where I already ran the install via Helm. https://github.com/solo-io/gloo/issues/4004#issuecomment-747839059

This didn't seem to work please see the comment for the result. I feel there is still a problem here.

Also, I am unsure on how to actually route the requests from the gateway to knative once it's finally installed properly.

Thanks!

P.S. Can you please reopen this issue?

@murphye I see you tried the advice listed in https://github.com/solo-io/gloo/issues/1544#issuecomment-547669320 , this advice is now dated and doesn't work in later versions of glooctl or helm with their default install flows.

For the knative service with IngressNotConfigured, can you check the logs of the knative pods to see if they give hints as to any reason for the status?

For routing from the gateway using virtual services to a knative service using virtual services, we would need to implement https://github.com/solo-io/gloo/issues/2410. This ticket would become a duplicate of that one.

Today, knative services leverage the standard Ingress resource in Gloo: https://github.com/solo-io/gloo/blob/01d04751f72c168e304977c4f67fdbcbf30232a9/projects/knative/pkg/translator/translate.go#L76

Ok, after trying the Gloo + Knative deployment again, I no longer have the IngressNotConfigured error. So that is a positive.

Here is the result of using the upstream gloo-system-knative-external-proxy-80. Based on another comment here https://github.com/solo-io/gloo/issues/1544#issuecomment-547669320 it seems like this could work, but in the end I get a connection error.

curl localhost:8081/all-pets                          
upstream connect error or disconnect/reset before headers. reset reason: connection failure

Output from glooctl proxy logs -f

[2020-12-19 03:50:36.735][35][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:837] [C46][S1580053177023262159] request headers complete (end_stream=true):
':authority', 'localhost:8081'
':path', '/all-pets'
':method', 'GET'
'user-agent', 'curl/7.64.1'
'accept', '*/*'

[2020-12-19 03:50:36.735][35][debug][http] [external/envoy/source/common/http/filter_manager.cc:721] [C46][S1580053177023262159] request end stream
[2020-12-19 03:50:36.735][35][debug][router] [external/envoy/source/common/router/router.cc:429] [C46][S1580053177023262159] cluster 'gloo-system-knative-external-proxy-80_gloo-system' match for URL '/all-pets'
[2020-12-19 03:50:36.735][35][debug][router] [external/envoy/source/common/router/router.cc:586] [C46][S1580053177023262159] router decoding headers:
':authority', 'petstore.default.example.com'
':path', '/api/pets'
':method', 'GET'
':scheme', 'http'
'user-agent', 'curl/7.64.1'
'accept', '*/*'
'x-forwarded-proto', 'http'
'x-request-id', '736025e0-3346-438f-ab58-b39af213ac38'
'x-envoy-expected-rq-timeout-ms', '15000'
'x-envoy-original-path', '/all-pets'

[2020-12-19 03:50:36.736][35][debug][pool] [external/envoy/source/common/http/conn_pool_base.cc:71] queueing stream due to no available connections
[2020-12-19 03:50:36.736][35][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:104] creating a new connection
[2020-12-19 03:50:36.736][35][debug][client] [external/envoy/source/common/http/codec_client.cc:39] [C47] connecting
[2020-12-19 03:50:36.736][35][debug][connection] [external/envoy/source/common/network/connection_impl.cc:769] [C47] connecting to 10.42.0.17:8080
[2020-12-19 03:50:36.737][35][debug][connection] [external/envoy/source/common/network/connection_impl.cc:785] [C47] connection in progress
[2020-12-19 03:50:36.737][35][debug][connection] [external/envoy/source/common/network/connection_impl.cc:634] [C47] delayed connection error: 111
[2020-12-19 03:50:36.738][35][debug][connection] [external/envoy/source/common/network/connection_impl.cc:203] [C47] closing socket: 0
[2020-12-19 03:50:36.738][35][debug][client] [external/envoy/source/common/http/codec_client.cc:96] [C47] disconnect. resetting 0 pending requests
[2020-12-19 03:50:36.738][35][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:314] [C47] client disconnected, failure reason: 
[2020-12-19 03:50:36.739][35][debug][router] [external/envoy/source/common/router/router.cc:1031] [C46][S1580053177023262159] upstream reset: reset reason: connection failure, transport failure reason: 
[2020-12-19 03:50:36.739][35][debug][http] [external/envoy/source/common/http/filter_manager.cc:805] [C46][S1580053177023262159] Sending local reply with details upstream_reset_before_response_started{connection failure}
[2020-12-19 03:50:36.739][35][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1435] [C46][S1580053177023262159] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '91'
'content-type', 'text/plain'
'date', 'Sat, 19 Dec 2020 03:50:36 GMT'
'server', 'envoy'

Here is the exact error from the log above:

[2020-12-19 03:50:36.739][35][debug][router] [external/envoy/source/common/router/router.cc:1031] [C46][S1580053177023262159] upstream reset: reset reason: connection failure, transport failure reason: 
[2020-12-19 03:50:36.739][35][debug][http] [external/envoy/source/common/http/filter_manager.cc:805] [C46][S1580053177023262159] Sending local reply with details upstream_reset_before_response_started{connection failure}

glooctl get upstream gloo-system-knative-external-proxy-80
+---------------------------------------+------------+----------+--------------------------------+
|               UPSTREAM                |    TYPE    |  STATUS  |            DETAILS             |
+---------------------------------------+------------+----------+--------------------------------+
| gloo-system-knative-external-proxy-80 | Kubernetes | Accepted | svc name:                      |
|                                       |            |          | knative-external-proxy         |
|                                       |            |          | svc namespace: gloo-system     |
|                                       |            |          | port:          80              |
|                                       |            |          |                                |
+---------------------------------------+------------+----------+--------------------------------+

From the logs above, it is pointing to the right IP, 10.42.0.17.

kubectl get pod -o wide -n gloo-system
NAME                                      READY   STATUS    RESTARTS   AGE     IP           NODE                       NOMINATED NODE   READINESS GATES
knative-external-proxy-f4f67644d-xwf88    1/1     Running   0          3h55m   10.42.0.17   k3d-k3s-default-server-0   <none>           <none>

From log above:

[2020-12-19 03:50:36.736][35][debug][connection] [external/envoy/source/common/network/connection_impl.cc:769] [C47] connecting to 10.42.0.17:8080

Running at 10.42.0.17:8080, so everything should be OK as seen below with the Pod definition... hmmm

kubectl describe pod knative-external-proxy-f4f67644d-xwf88 -n gloo-system
Name:         knative-external-proxy-f4f67644d-xwf88
Namespace:    gloo-system
Priority:     0
Node:         k3d-k3s-default-server-0/172.18.0.2
Start Time:   Fri, 18 Dec 2020 16:15:32 -0800
Labels:       gloo=knative-external-proxy
              pod-template-hash=f4f67644d
Annotations:  <none>
Status:       Running
IP:           10.42.0.17
IPs:
  IP:           10.42.0.17
Controlled By:  ReplicaSet/knative-external-proxy-f4f67644d
Containers:
  knative-external-proxy:
    Container ID:  containerd://9fa5c84d25a3e1dc9aabc8b58db66ef68b131d5ace39ecb641a2b88a55bdce10
    Image:         quay.io/solo-io/gloo-envoy-wrapper:1.5.14
    Image ID:      quay.io/solo-io/gloo-envoy-wrapper@sha256:6df86b35d2bbfec779a6cb52e212527547a8bb109d966488810c8379877a9d16
    Ports:         8080/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      --disable-hot-restart
    State:          Running
      Started:      Fri, 18 Dec 2020 16:15:55 -0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAMESPACE:  gloo-system (v1:metadata.namespace)
      POD_NAME:       knative-external-proxy-f4f67644d-xwf88 (v1:metadata.name)
    Mounts:
      /etc/envoy from envoy-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-v26b9 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  envoy-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      knative-external-proxy-config
    Optional:  false
  default-token-v26b9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-v26b9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

Getting some warnings/errors in the knative-external-proxy logs. Also it seems to be rejecting any connections on port 8080, which explains the connection failure error in the logs in the previous comment.

kubectl logs knative-external-proxy-f4f67644d-xwf88 -n gloo-system
[2020-12-19 00:15:55.697][8][info][main] [external/envoy/source/server/server.cc:305] initializing epoch 0 (base id=0, hot restart version=disabled)
[2020-12-19 00:15:55.697][8][info][main] [external/envoy/source/server/server.cc:307] statically linked extensions:
[2020-12-19 00:15:55.697][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2020-12-19 00:15:55.697][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2020-12-19 00:15:55.697][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.http.cache: envoy.extensions.http.cache.simple
[2020-12-19 00:15:55.697][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.udp_listeners: raw_udp_listener
[2020-12-19 00:15:55.697][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.bootstrap: envoy.extensions.network.socket_interface.default_socket_interface
[2020-12-19 00:15:55.697][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.dubbo_proxy.route_matchers: default
[2020-12-19 00:15:55.697][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.udp_packet_writers: udp_default_writer
[2020-12-19 00:15:55.698][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.resolvers: envoy.ip
[2020-12-19 00:15:55.698][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2020-12-19 00:15:55.698][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.tcp_grpc, envoy.file_access_log, envoy.http_grpc_access_log, envoy.tcp_grpc_access_log
[2020-12-19 00:15:55.698][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2020-12-19 00:15:55.698][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.guarddog_actions: envoy.watchdog.profile_action
[2020-12-19 00:15:55.698][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.compression.decompressor: envoy.compression.gzip.decompressor
[2020-12-19 00:15:55.698][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2020-12-19 00:15:55.698][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.dubbo_proxy.protocols: dubbo
[2020-12-19 00:15:55.698][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2020-12-19 00:15:55.698][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.upstreams: envoy.filters.connection_pools.http.generic, envoy.filters.connection_pools.http.http, envoy.filters.connection_pools.http.tcp
[2020-12-19 00:15:55.698][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2020-12-19 00:15:55.698][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.compression.compressor: envoy.compression.gzip.compressor
[2020-12-19 00:15:55.698][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2020-12-19 00:15:55.699][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.health_checkers: envoy.health_checkers.redis
[2020-12-19 00:15:55.699][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2020-12-19 00:15:55.699][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2020-12-19 00:15:55.699][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.postgres_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.rocketmq_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2020-12-19 00:15:55.699][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2020-12-19 00:15:55.699][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2020-12-19 00:15:55.699][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.stats_sinks: envoy.dog_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.statsd
[2020-12-19 00:15:55.699][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.thrift_proxy.filters: envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2020-12-19 00:15:55.699][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.gzip, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.lua, envoy.filters.http.oauth, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.squash, envoy.filters.http.tap, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash, io.solo.aws_lambda, io.solo.nats_streaming, io.solo.transformation
[2020-12-19 00:15:55.699][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2020-12-19 00:15:55.699][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2020-12-19 00:15:55.699][8][info][main] [external/envoy/source/server/server.cc:309]   envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2020-12-19 00:15:55.716][8][info][main] [external/envoy/source/server/server.cc:325] HTTP header map info:
[2020-12-19 00:15:55.717][8][warning][runtime] [external/envoy/source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2020-12-19 00:15:55.717][8][warning][runtime] [external/envoy/source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2020-12-19 00:15:55.718][8][warning][runtime] [external/envoy/source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2020-12-19 00:15:55.719][8][warning][runtime] [external/envoy/source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2020-12-19 00:15:55.719][8][info][main] [external/envoy/source/server/server.cc:328]   request header map: 600 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-method,authorization,cache-control,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-proto,x-ot-span-context,x-request-id
[2020-12-19 00:15:55.719][8][info][main] [external/envoy/source/server/server.cc:328]   request trailer map: 128 bytes: 
[2020-12-19 00:15:55.719][8][info][main] [external/envoy/source/server/server.cc:328]   response header map: 424 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2020-12-19 00:15:55.719][8][info][main] [external/envoy/source/server/server.cc:328]   response trailer map: 152 bytes: grpc-message,grpc-status
[2020-12-19 00:15:55.722][8][info][main] [external/envoy/source/server/server.cc:448] admin address: 127.0.0.1:19000
[2020-12-19 00:15:55.726][8][info][main] [external/envoy/source/server/server.cc:583] runtime: layers:
  - name: base
    static_layer:
      {}
  - name: admin
    admin_layer:
      {}
[2020-12-19 00:15:55.726][8][info][config] [external/envoy/source/server/configuration_impl.cc:95] loading tracing configuration
[2020-12-19 00:15:55.726][8][info][config] [external/envoy/source/server/configuration_impl.cc:70] loading 0 static secret(s)
[2020-12-19 00:15:55.726][8][info][config] [external/envoy/source/server/configuration_impl.cc:76] loading 2 cluster(s)
[2020-12-19 00:15:55.734][8][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-12-19 00:15:55.734][8][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:63] Unable to establish new stream
[2020-12-19 00:15:55.734][8][info][config] [external/envoy/source/server/configuration_impl.cc:80] loading 0 listener(s)
[2020-12-19 00:15:55.734][8][info][config] [external/envoy/source/server/configuration_impl.cc:121] loading stats sink configuration
[2020-12-19 00:15:55.735][8][info][main] [external/envoy/source/server/server.cc:679] starting main dispatch loop
[2020-12-19 00:15:55.739][8][info][runtime] [external/envoy/source/common/runtime/runtime_impl.cc:421] RTDS has finished initialization
[2020-12-19 00:15:55.739][8][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:174] cm init: initializing cds
[2020-12-19 00:15:55.739][8][warning][main] [external/envoy/source/server/server.cc:565] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
[2020-12-19 00:15:57.550][8][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
[2020-12-19 00:15:58.958][8][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
[2020-12-19 00:16:03.298][8][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:64] cds: add 0 cluster(s), remove 2 cluster(s)
[2020-12-19 00:16:03.298][8][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:178] cm init: all clusters initialized
[2020-12-19 00:16:03.299][8][info][main] [external/envoy/source/server/server.cc:660] all clusters initialized. initializing init manager
[2020-12-19 00:16:03.300][8][info][config] [external/envoy/source/server/listener_manager_impl.cc:888] all dependencies initialized. starting workers
[2020-12-19 00:31:02.640][8][info][main] [external/envoy/source/server/drain_manager_impl.cc:70] shutting down parent after drain

This one looks interesting:

[2020-12-19 00:15:58.958][8][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure

Update: After installing Knative without Gateway, knative-external-proxy contains the same warnings/errors but remains functional. So this seems to be unrelated to the connection issue.

I ran two separate Gloo installs (Knative and Gateway+Knative) and want to check connectivity to knative-external-proxy 8080 in each scenario. I sh into the ingress pods on each and run wget.

Knative knative-external-proxy

I am able to connect to knative-external-proxy 8080 and return a 404, which is the expected result. This is OK.

kubectl exec --stdin --tty ingress-6f95fd7dc-67rwj -n gloo-system -- /bin/sh
/ $ wget 10.42.0.12:8080
Connecting to 10.42.0.12:8080 (10.42.0.12:8080)
wget: server returned error: HTTP/1.1 404 Not Found

Gateway+Knative knative-external-proxy

In this scenario, the connection is refused from knative-external-proxy 8080. This is bad, and I am not yet sure why this is happening.

kubectl exec --stdin --tty ingress-6f7f8bdcd9-fpqbz -n gloo-system -- /bin/sh
/ $ wget 10.42.0.6:8080
Connecting to 10.42.0.6:8080 (10.42.0.6:8080)
wget: can't connect to remote host (10.42.0.6): Connection refused

There seems to be some deeper issue where the knative proxy are possibly not being configured correctly in the Gateway+Knative scenario. The logs do not have any obvious errors that would cause this issue.

@kdorosh Do you have any thoughts on why knative-external-proxy is not responding when installed in tandem with the gateway? See the previous comment. Is there a possibility that configuration is not being pushed down to the proxy?

Thanks!

This issue has been marked as stale because of no activity in the last 180 days. It will be closed in the next 180 days unless it is tagged "no stalebot" or other activity occurs.

This issue has been closed due to no activity in the last 12 months.

solo-io / gloo

Gloo Edge doesn't support a hybrid of gateway and knative profiles #4004