Ingress Gateway: no routing, 404 Not Found

pkrolikowski commented 3 years ago

Hi Team!

Overview of the Issue

I'm trying to setup ingress gateway with routing based on Hosts field (or any simple setup, including wildcard on Name field). Envoy always returns 404 Not Found

Reproduction Steps

Create ingress gw. using this config:

"Kind": "ingress-gateway",
"Name": "us-east-ingress",
"TLS": {
    "Enabled": false
},
"Listeners": [
    {
        "Port": 80,
        "Protocol": "http",
        "Services": [
            {
                "Name": "pk-server",
                "Hosts": [
                    "pk-api.example.com",
                    "pk-server.ingress.consul",
                    "*.consul"
                ]
            }
        ]
    }
],
"CreateIndex": 130475,
"ModifyIndex": 131469
}

Register consul service with connect proxy

{
"service": {
"port": 5002,
"name": "pk-server",
"connect": { "sidecar_service": {} }
}
}

Set proxy-defaults:

Kind      = "proxy-defaults"
Name      = "global"
Config {
protocol = "http"
}

Run connect proxy: consul connect envoy -gateway=ingress -register -service='us-east-ingress' -address 'xx.xx.xx.xx:80' -token='<consul_token>' -admin-bind='xx.xx.xx.xx:1900' -omit-deprecated-tags

Try curl to the service:

curl -v -H "Host: pk-server.ingress.consul" <private_ip>
*   Trying xx.xx.xx.xx:80...
* TCP_NODELAY set
* Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 80 (#0)
> GET / HTTP/1.1
> Host: pk-server.ingress.consul
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< date: Wed, 09 Dec 2020 14:26:12 GMT
< server: envoy
< content-length: 0
< 
* Connection #0 to host xx.xx.xx.xx left intact

curl using any value from Hosts also failing.

Debugging

Name resolution looks good:

dig +short @127.0.0.1 -p 8600 pk-server.ingress.consul SRV
1 1 80 0af00037.addr.pk.consul.

connect intentions

``` [ { "CreatedAt": "2020-12-09T15:45:12.752819454Z", "UpdatedAt": "2020-12-09T15:45:12.752819454Z", "ID": "ff656e3b-cdb8-cbbd-fb53-c875a8895564", "SourceNS": "default", "SourceName": "us-east-ingress", "DestinationNS": "default", "DestinationName": "pk-server", "SourceType": "consul", "Action": "allow", "Precedence": 9, "Hash": "sY36OcnUwSTzuD6265Sl5fyWZRIvD4zKC4k84Duk3rI=", "CreateIndex": 132413, "ModifyIndex": 132413 }, { "CreatedAt": "2020-11-30T13:38:55.49354711Z", "UpdatedAt": "2020-11-30T13:38:55.49354711Z", "ID": "e46aeb04-1fb6-b002-0985-044169c0bb87", "Description": "all-1-all", "SourceNS": "default", "SourceName": "*", "DestinationNS": "default", "DestinationName": "*", "SourceType": "consul", "Action": "allow", "Precedence": 5, "Hash": "kYyv7O318vAEgLeAeILxm32zxfYX5bpKLJSgH7Ze5xs=", "CreateIndex": 279, "ModifyIndex": 279 } ] ```

services associated with an ingress gateway

``` [ { "Gateway": { "Name": "us-east-ingress" }, "Service": { "Name": "pk-server" }, "GatewayKind": "ingress-gateway", "Port": 80, "Protocol": "http", "Hosts": [ "pk-api.stobworg.co", "pk-server.ingress.consul", "*.consul" ], "CreateIndex": 132076, "ModifyIndex": 132076 } ] ```

envoy dynamic route config

``` "dynamic_route_configs": [ { "version_info": "00000001", "route_config": { "@type": "type.googleapis.com/envoy.api.v2.RouteConfiguration", "name": "80", "virtual_hosts": [ { "name": "pk-server", "domains": [ "pk-api.example.com", "pk-server.ingress.consul", "*.consul", "pk-api.example.com:80", "pk-server.ingress.consul:80", "*.consul:80" ], "routes": [ { "match": { "prefix": "/" }, "route": { "cluster": "pk-server.default.pk.internal.fa8428e8-a896-b606-3983-953731abba55.consul" } } ] } ], "validate_clusters": true }, "last_updated": "2020-12-09T14:45:12.238Z" } ] } ```

I am able to connect to service pk-server from other service via consul connect proxy

Consul info for both Client and Server

Client and server info

``` agent: check_monitors = 0 check_ttls = 0 checks = 8 services = 8 build: prerelease = revision = a417fe51 version = 1.9.0 consul: acl = enabled bootstrap = false known_datacenters = 1 leader = false leader_addr = xx.xx.xx.xx:8300 server = true raft: applied_index = 131326 commit_index = 131326 fsm_pending = 0 last_contact = 38.86848ms last_log_index = 131326 last_log_term = 6 last_snapshot_index = 131095 last_snapshot_term = 6 latest_configuration = [{Suffrage:Voter ID:703fc5a4-6520-f587-82f8-26c75c49d508 Address:xx.xx.xx.xx:8300} {Suffrage:Voter ID:fc705b76-eebe-d5b0-6670-876eabdfd6a4 Address:xx.xx.xx.xx:8300} {Suffrage:Voter ID:f8e3faf6-da8f-e021-df69-5f8729aaa584 Address:xx.xx.xx.xx:8300}] latest_configuration_index = 0 num_peers = 2 protocol_version = 3 protocol_version_max = 3 protocol_version_min = 0 snapshot_version_max = 1 snapshot_version_min = 0 state = Follower term = 6 runtime: arch = amd64 cpu_count = 1 goroutines = 202 max_procs = 1 os = linux version = go1.15.5 serf_lan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 6 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 11 members = 4 query_queue = 0 query_time = 1 serf_wan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 1 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 12 members = 3 query_queue = 0 query_time = 1 ```

Operating system and Environment details

OS

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:    20.04
Codename:   focal

envoy

envoy  version: 8fb3cb86082b17144a80402f5367ae65f06083bd/1.16.0/clean-getenvoy-a5345f6-envoy/RELEASE/BoringSSL

Consul

Consul v1.9.0
Revision a417fe510

HofmannZ commented 3 years ago

@pkrolikowski - Did you find a solution?

neonthe1way commented 3 years ago

@pkrolikowski / @HofmannZ - Did you find a solution? I am facing similar issue with HTTP routing. Any work around will also be helpful

HofmannZ commented 3 years ago

@neonthe1way yes, in our case we needed to add the correct host headers on the request so that the ingress gateway knew where to route to.

In the end we never ended up using it in production tho.

mario45211 commented 2 years ago

@pkrolikowski / @HofmannZ - Did you find a solution? I am facing similar issue with HTTP routing. Any work around will also be helpful

As workaround you can use wildcard for Hosts header, which accept all its values like:

//...
 Services = [
     {
       Name = "public-api"
       Hosts = ["*"]
     }
   ]

hossain666 commented 1 month ago

troubleshoot_server.sh

hashicorp / consul