search

holos-run / holos

Holos - The Holistic platform manager
https://holos.run
Apache License 2.0
1 stars 0 forks source link

404 NR - Fix Gateway default again #170

Closed jeffmccune closed 2 months ago

jeffmccune commented 4 months ago

We are running into https://github.com/istio/istio/issues/9429 again with our current Gateway configuration.

For example, hitting https://jeff.app.dev.k2.holos.run/ first causes https://app.dev.k2.holos.run/ to return a 404 NR.

The istio logs show the requested server name different from the authority:

{
  "response_flags": "NR",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
  "response_code": 404,
  "path": "/ui/platform/018f36fb-e3ff-7f7f-a5d1-7ca2bf499e94",
  "upstream_cluster": null,
  "route_name": null,
  "requested_server_name": "app.dev.k2.holos.run",
  "start_time": "2024-05-08T20:16:31.162Z",
  "request_id": "f4393bc8-ba52-4a22-a959-16d6a919336a",
  "x_forwarded_for": "192.168.2.21",
  "connection_termination_details": null,
  "downstream_local_address": "65.102.23.41:443",
  "upstream_service_time": null,
  "authority": "jeff.app.dev.k2.holos.run",
  "upstream_host": null,
  "protocol": "HTTP/2",
  "duration": 0,
  "bytes_received": 0,
  "downstream_remote_address": "192.168.2.21:57968",
  "upstream_transport_failure_reason": null,
  "method": "GET",
  "upstream_local_address": null,
  "bytes_sent": 0,
  "response_code_details": "route_not_found"
}

The recommended fix is here:

[!NOTE] You can avoid this problem by configuring a single wildcard Gateway, instead of two (gw1 and gw2). Then, simply bind both VirtualServices to it like this:

  • Gateway configuration gw with host *.test.com, selector istio: ingressgateway, and TLS config using gateway's mounted (wildcard) cert
  • VirtualService configuration vs1 with host service1.test.com and gateway gw
  • VirtualService configuration vs2 with host service2.test.com and gateway gw

We are not following this in 0.74.0:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  annotations:
    holos.run/description: ""
  creationTimestamp: "2024-03-11T23:30:34Z"
  generation: 3
  labels:
    kustomize.toolkit.fluxcd.io/name: prod-mesh-gateway
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: default
  namespace: istio-ingress
  resourceVersion: "18353606"
  uid: 4c10ef30-4f7c-45f2-9e59-792129dcd506
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - istio-ingress/httpbin.k2.ois.run
    port:
      name: https-istio-ingress-httpbin
      number: 443
      protocol: HTTPS
    tls:
      credentialName: k2-httpbin
      mode: SIMPLE
  - hosts:
    - prod-platform/argocd.ois.run
    - prod-platform-system/argocd.ois.run
    port:
      name: https-argocd-ois-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: argocd.ois.run
      mode: SIMPLE
  - hosts:
    - prod-platform/grafana.ois.run
    - prod-platform-system/grafana.ois.run
    port:
      name: https-grafana-ois-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: grafana.ois.run
      mode: SIMPLE
  - hosts:
    - prod-platform/prometheus.ois.run
    - prod-platform-system/prometheus.ois.run
    port:
      name: https-prometheus-ois-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: prometheus.ois.run
      mode: SIMPLE
  - hosts:
    - dev-platform/argocd.dev.ois.run
    - dev-platform-system/argocd.dev.ois.run
    port:
      name: https-argocd-dev-ois-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: argocd.dev.ois.run
      mode: SIMPLE
  - hosts:
    - dev-platform/grafana.dev.ois.run
    - dev-platform-system/grafana.dev.ois.run
    port:
      name: https-grafana-dev-ois-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: grafana.dev.ois.run
      mode: SIMPLE
  - hosts:
    - dev-platform/prometheus.dev.ois.run
    - dev-platform-system/prometheus.dev.ois.run
    port:
      name: https-prometheus-dev-ois-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: prometheus.dev.ois.run
      mode: SIMPLE
  - hosts:
    - prod-platform/argocd.k2.ois.run
    - prod-platform-system/argocd.k2.ois.run
    port:
      name: https-argocd-k2-ois-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: argocd.ois.run
      mode: SIMPLE
  - hosts:
    - prod-platform/grafana.k2.ois.run
    - prod-platform-system/grafana.k2.ois.run
    port:
      name: https-grafana-k2-ois-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: grafana.ois.run
      mode: SIMPLE
  - hosts:
    - prod-platform/prometheus.k2.ois.run
    - prod-platform-system/prometheus.k2.ois.run
    port:
      name: https-prometheus-k2-ois-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: prometheus.ois.run
      mode: SIMPLE
  - hosts:
    - dev-platform/argocd.dev.k2.ois.run
    - dev-platform-system/argocd.dev.k2.ois.run
    port:
      name: https-argocd-dev-k2-ois-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: argocd.dev.ois.run
      mode: SIMPLE
  - hosts:
    - dev-platform/grafana.dev.k2.ois.run
    - dev-platform-system/grafana.dev.k2.ois.run
    port:
      name: https-grafana-dev-k2-ois-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: grafana.dev.ois.run
      mode: SIMPLE
  - hosts:
    - dev-platform/prometheus.dev.k2.ois.run
    - dev-platform-system/prometheus.dev.k2.ois.run
    port:
      name: https-prometheus-dev-k2-ois-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: prometheus.dev.ois.run
      mode: SIMPLE
  - hosts:
    - prod-holos/app.holos.run
    - prod-holos-system/app.holos.run
    port:
      name: https-app-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: app.holos.run
      mode: SIMPLE
  - hosts:
    - prod-holos/provision.holos.run
    - prod-holos-system/provision.holos.run
    port:
      name: https-provision-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: provision.holos.run
      mode: SIMPLE
  - hosts:
    - prod-holos/nats.holos.run
    - prod-holos-system/nats.holos.run
    port:
      name: https-nats-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: nats.holos.run
      mode: SIMPLE
  - hosts:
    - dev-holos/app.dev.holos.run
    - dev-holos-system/app.dev.holos.run
    port:
      name: https-app-dev-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: app.dev.holos.run
      mode: SIMPLE
  - hosts:
    - dev-holos/provision.dev.holos.run
    - dev-holos-system/provision.dev.holos.run
    port:
      name: https-provision-dev-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: provision.dev.holos.run
      mode: SIMPLE
  - hosts:
    - dev-holos/nats.dev.holos.run
    - dev-holos-system/nats.dev.holos.run
    port:
      name: https-nats-dev-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: nats.dev.holos.run
      mode: SIMPLE
  - hosts:
    - jeff-holos/jeff.app.dev.holos.run
    - dev-holos-system/jeff.app.dev.holos.run
    port:
      name: https-jeff-app-dev-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: app.dev.holos.run
      mode: SIMPLE
  - hosts:
    - jeff-holos/jeff.provision.dev.holos.run
    - dev-holos-system/jeff.provision.dev.holos.run
    port:
      name: https-jeff-provision-dev-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: provision.dev.holos.run
      mode: SIMPLE
  - hosts:
    - jeff-holos/jeff.nats.dev.holos.run
    - dev-holos-system/jeff.nats.dev.holos.run
    port:
      name: https-jeff-nats-dev-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: nats.dev.holos.run
      mode: SIMPLE
  - hosts:
    - gary-holos/gary.app.dev.holos.run
    - dev-holos-system/gary.app.dev.holos.run
    port:
      name: https-gary-app-dev-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: app.dev.holos.run
      mode: SIMPLE
  - hosts:
    - gary-holos/gary.provision.dev.holos.run
    - dev-holos-system/gary.provision.dev.holos.run
    port:
      name: https-gary-provision-dev-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: provision.dev.holos.run
      mode: SIMPLE
  - hosts:
    - gary-holos/gary.nats.dev.holos.run
    - dev-holos-system/gary.nats.dev.holos.run
    port:
      name: https-gary-nats-dev-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: nats.dev.holos.run
      mode: SIMPLE
  - hosts:
    - nate-holos/nate.app.dev.holos.run
    - dev-holos-system/nate.app.dev.holos.run
    port:
      name: https-nate-app-dev-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: app.dev.holos.run
      mode: SIMPLE
  - hosts:
    - nate-holos/nate.provision.dev.holos.run
    - dev-holos-system/nate.provision.dev.holos.run
    port:
      name: https-nate-provision-dev-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: provision.dev.holos.run
      mode: SIMPLE
  - hosts:
    - nate-holos/nate.nats.dev.holos.run
    - dev-holos-system/nate.nats.dev.holos.run
    port:
      name: https-nate-nats-dev-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: nats.dev.holos.run
      mode: SIMPLE
  - hosts:
    - prod-holos/app.k2.holos.run
    - prod-holos-system/app.k2.holos.run
    port:
      name: https-app-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: app.holos.run
      mode: SIMPLE
  - hosts:
    - prod-holos/provision.k2.holos.run
    - prod-holos-system/provision.k2.holos.run
    port:
      name: https-provision-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: provision.holos.run
      mode: SIMPLE
  - hosts:
    - prod-holos/nats.k2.holos.run
    - prod-holos-system/nats.k2.holos.run
    port:
      name: https-nats-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: nats.holos.run
      mode: SIMPLE
  - hosts:
    - dev-holos/app.dev.k2.holos.run
    - dev-holos-system/app.dev.k2.holos.run
    port:
      name: https-app-dev-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: app.dev.holos.run
      mode: SIMPLE
  - hosts:
    - dev-holos/provision.dev.k2.holos.run
    - dev-holos-system/provision.dev.k2.holos.run
    port:
      name: https-provision-dev-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: provision.dev.holos.run
      mode: SIMPLE
  - hosts:
    - dev-holos/nats.dev.k2.holos.run
    - dev-holos-system/nats.dev.k2.holos.run
    port:
      name: https-nats-dev-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: nats.dev.holos.run
      mode: SIMPLE
  - hosts:
    - jeff-holos/jeff.app.dev.k2.holos.run
    - dev-holos-system/jeff.app.dev.k2.holos.run
    port:
      name: https-jeff-app-dev-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: app.dev.holos.run
      mode: SIMPLE
  - hosts:
    - jeff-holos/jeff.provision.dev.k2.holos.run
    - dev-holos-system/jeff.provision.dev.k2.holos.run
    port:
      name: https-jeff-provision-dev-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: provision.dev.holos.run
      mode: SIMPLE
  - hosts:
    - jeff-holos/jeff.nats.dev.k2.holos.run
    - dev-holos-system/jeff.nats.dev.k2.holos.run
    port:
      name: https-jeff-nats-dev-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: nats.dev.holos.run
      mode: SIMPLE
  - hosts:
    - gary-holos/gary.app.dev.k2.holos.run
    - dev-holos-system/gary.app.dev.k2.holos.run
    port:
      name: https-gary-app-dev-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: app.dev.holos.run
      mode: SIMPLE
  - hosts:
    - gary-holos/gary.provision.dev.k2.holos.run
    - dev-holos-system/gary.provision.dev.k2.holos.run
    port:
      name: https-gary-provision-dev-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: provision.dev.holos.run
      mode: SIMPLE
  - hosts:
    - gary-holos/gary.nats.dev.k2.holos.run
    - dev-holos-system/gary.nats.dev.k2.holos.run
    port:
      name: https-gary-nats-dev-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: nats.dev.holos.run
      mode: SIMPLE
  - hosts:
    - nate-holos/nate.app.dev.k2.holos.run
    - dev-holos-system/nate.app.dev.k2.holos.run
    port:
      name: https-nate-app-dev-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: app.dev.holos.run
      mode: SIMPLE
  - hosts:
    - nate-holos/nate.provision.dev.k2.holos.run
    - dev-holos-system/nate.provision.dev.k2.holos.run
    port:
      name: https-nate-provision-dev-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: provision.dev.holos.run
      mode: SIMPLE
  - hosts:
    - nate-holos/nate.nats.dev.k2.holos.run
    - dev-holos-system/nate.nats.dev.k2.holos.run
    port:
      name: https-nate-nats-dev-k2-holos-run
      number: 443
      protocol: HTTPS
    tls:
      credentialName: nats.dev.holos.run
      mode: SIMPLE
jeffmccune commented 4 months ago

Note this is only a problem for dev which uses a wildcard cert:

k get secret -n istio-ingress app.dev.holos.run -o json \
  | jq --exit-status '.data | map_values(@base64d)' \
  | jq '."tls.crt"' -r \
  | openssl x509 -text -noout -in - \
  | grep app.dev.k2.holos.run
jeffmccune commented 2 months ago

Pretty sure this got fixed for aws2 in https://github.com/holos-run/holos-infra/commit/09a7709c0b8c4c67bc3ecd98845151304577cc0d