search

solo-io / gloo

The Feature-rich, Kubernetes-native, Next-Generation API Gateway Built on Envoy
https://docs.solo.io/
Apache License 2.0
4.09k stars 442 forks source link

issue with gateway-proxy #2209

Closed zetaab closed 4 years ago

zetaab commented 4 years ago

I installed gloo using glooctl install gateway. However, this whole stuff does not work at all.

Kubernetes service:

% kubectl describe svc gateway-proxy
Name:                     gateway-proxy
Namespace:                gloo-system
Labels:                   app=gloo
                          gateway-proxy-id=gateway-proxy
                          gloo=gateway-proxy
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"gloo","gateway-proxy-id":"gateway-proxy","gloo":"gateway...
Selector:                 gateway-proxy-id=gateway-proxy,gateway-proxy=live
Type:                     LoadBalancer
IP:                       100.64.174.186
LoadBalancer Ingress:     10.222.138.206
Port:                     http  80/TCP
TargetPort:               8080/TCP
NodePort:                 http  32175/TCP
Endpoints:                100.110.158.136:8080
Port:                     https  443/TCP
TargetPort:               8443/TCP
NodePort:                 https  30255/TCP
Endpoints:                100.110.158.136:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                Age   From                Message
  ----    ------                ----  ----                -------
  Normal  EnsuringLoadBalancer  17m   service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   15m   service-controller  Ensured load balancer

However, when I try to connect those endpoints from inside cluster. Those does not answer anything:

% kubectl exec -it curltest /bin/sh
/ # curl 100.110.158.136:8080
curl: (7) Failed to connect to 100.110.158.136 port 8080: Connection refused
/ # curl 100.110.158.136:8443
curl: (7) Failed to connect to 100.110.158.136 port 8443: Connection refused

When I list ports listening in gateway-proxy:

% kubectl exec -it gateway-proxy-7579d7f5c8-fzzvs /bin/sh
/ # netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:19000         0.0.0.0:*               LISTEN
tcp        0      0 100.110.158.136:48528   100.64.47.226:9977      ESTABLISHED
tcp        0      0 100.110.158.136:36234   100.64.47.226:9966      ESTABLISHED
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path

log file for gateway-proxy does not say anything bad 

% kubectl logs gateway-proxy-7579d7f5c8-fzzvs
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:251] initializing epoch 0 (hot restart version=disabled)
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:253] statically linked extensions:
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   access_loggers: envoy.file_access_log, envoy.http_grpc_access_log, envoy.tcp_grpc_access_log
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   dubbo_proxy.filters: envoy.filters.dubbo.router
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   dubbo_proxy.protocols: dubbo
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   dubbo_proxy.route_matchers: default
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   dubbo_proxy.serializers: dubbo.hessian2
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_stats, envoy.filters.http.header_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.original_src, envoy.filters.http.rbac, envoy.filters.http.tap, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash, io.solo.aws_lambda, io.solo.nats_streaming, io.solo.transformation
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   filters.listener: envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.dubbo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.rbac, envoy.filters.network.sni_cluster, envoy.filters.network.thrift_proxy, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   filters.udp_listener: envoy.filters.udp_listener.udp_proxy
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   health_checkers: envoy.health_checkers.redis
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   resolvers: envoy.ip
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.previous_hosts
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   retry_priorities: envoy.retry_priorities.previous_priorities
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   stats_sinks: envoy.dog_statsd, envoy.metrics_service, envoy.stat_sinks.hystrix, envoy.statsd
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   thrift_proxy.filters: envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   thrift_proxy.transports: auto, framed, header, unframed
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.opencensus, envoy.tracers.xray, envoy.zipkin
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2020-01-16 11:07:41.569][6][info][main] [external/envoy/source/server/server.cc:255]   udp_listeners: raw_udp_listener
[2020-01-16 11:07:41.574][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:356] Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2020-01-16 11:07:41.574][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:356] Using deprecated option 'envoy.config.metrics.v2.StatsSink.config' from file stats.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2020-01-16 11:07:41.574][6][info][main] [external/envoy/source/server/server.cc:322] admin address: 127.0.0.1:19000
[2020-01-16 11:07:41.575][6][info][main] [external/envoy/source/server/server.cc:441] runtime: layers:
  - name: base
    static_layer:
      {}
  - name: admin
    admin_layer:
      {}
[2020-01-16 11:07:41.575][6][info][config] [external/envoy/source/server/configuration_impl.cc:60] loading 0 static secret(s)
[2020-01-16 11:07:41.575][6][info][config] [external/envoy/source/server/configuration_impl.cc:66] loading 4 cluster(s)
[2020-01-16 11:07:41.581][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[2020-01-16 11:07:41.581][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-01-16 11:07:41.581][6][info][config] [external/envoy/source/server/configuration_impl.cc:70] loading 1 listener(s)
[2020-01-16 11:07:41.582][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:356] Using deprecated option 'envoy.config.filter.network.http_connection_manager.v2.HttpFilter.config' from file http_connection_manager.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2020-01-16 11:07:41.584][6][info][config] [external/envoy/source/server/configuration_impl.cc:95] loading tracing configuration
[2020-01-16 11:07:41.584][6][info][config] [external/envoy/source/server/configuration_impl.cc:115] loading stats sink configuration
[2020-01-16 11:07:41.584][6][info][main] [external/envoy/source/server/server.cc:532] starting main dispatch loop
[2020-01-16 11:07:41.595][6][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:163] cm init: initializing cds
[2020-01-16 11:07:56.595][6][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:167] cm init: all clusters initialized
[2020-01-16 11:07:56.595][6][info][main] [external/envoy/source/server/server.cc:511] all clusters initialized. initializing init manager
[2020-01-16 11:08:11.597][6][info][config] [external/envoy/source/server/listener_manager_impl.cc:661] all dependencies initialized. starting workers
[2020-01-16 11:23:11.597][6][info][main] [external/envoy/source/server/drain_manager_impl.cc:65] shutting down parent after drain

gateways:

% kubectl get gateway -o yaml
apiVersion: v1
items:
- apiVersion: gateway.solo.io/v1
  kind: Gateway
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"gateway.solo.io/v1","kind":"Gateway","metadata":{"annotations":{},"labels":{"app":"gloo"},"name":"gateway-proxy","namespace":"gloo-system"},"spec":{"bindAddress":"::","bindPort":8080,"httpGateway":{},"proxyNames":["gateway-proxy"],"ssl":false,"useProxyProto":false}}
    creationTimestamp: "2020-01-16T11:06:24Z"
    generation: 1
    labels:
      app: gloo
    name: gateway-proxy
    namespace: gloo-system
    resourceVersion: "393884"
    selfLink: /apis/gateway.solo.io/v1/namespaces/gloo-system/gateways/gateway-proxy
    uid: 4c9d183a-65f2-4053-91ae-898456d8aae3
  spec:
    bindAddress: '::'
    bindPort: 8080
    httpGateway: {}
    proxyNames:
    - gateway-proxy
    ssl: false
    useProxyProto: false
- apiVersion: gateway.solo.io/v1
  kind: Gateway
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"gateway.solo.io/v1","kind":"Gateway","metadata":{"annotations":{},"labels":{"app":"gloo"},"name":"gateway-proxy-ssl","namespace":"gloo-system"},"spec":{"bindAddress":"::","bindPort":8443,"httpGateway":{},"proxyNames":["gateway-proxy"],"ssl":true,"useProxyProto":false}}
    creationTimestamp: "2020-01-16T11:06:22Z"
    generation: 1
    labels:
      app: gloo
    name: gateway-proxy-ssl
    namespace: gloo-system
    resourceVersion: "393871"
    selfLink: /apis/gateway.solo.io/v1/namespaces/gloo-system/gateways/gateway-proxy-ssl
    uid: db1dc380-e2b1-4cf1-bd43-8e757cee2df9
  spec:
    bindAddress: '::'
    bindPort: 8443
    httpGateway: {}
    proxyNames:
    - gateway-proxy
    ssl: true
    useProxyProto: false
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Now the only question is: why gateway-proxy does not have 8080 and 8443 ports listening? I did also new cluster and tried to install gloo as ingress controller - same problem. I cannot find way to install gloo that I could follow up "hello world" examples.

and all pods are btw healthy

% kubectl get pods
NAME                             READY   STATUS      RESTARTS   AGE
curltest                         1/1     Running     0          11m
discovery-54648cb47b-fdxph       1/1     Running     0          21m
gateway-55d954844f-kpl4w         1/1     Running     0          21m
gateway-certgen-56ghj            0/1     Completed   0          21m
gateway-proxy-7579d7f5c8-fzzvs   1/1     Running     0          19m
gloo-5b98c8f9c5-2gmnz            1/1     Running     0          21m
christian-posta commented 4 years ago

Did you create a route? By default, no listeners are opened until you have a valid routing config.

See https://docs.solo.io/gloo/latest/introduction/faq/#why-are-the-ports-on-my-gloo-gateway-proxy-not-opened

christian-posta commented 4 years ago

Let us know what happens after you create a route. This should resolve the ports not listening. I've also added some updates to the docs to make this more clear.

https://github.com/solo-io/gloo/pull/2223

rickducott commented 4 years ago

This appears to be resolved, let us know if you continue to experience problems.