search

cloudfoundry / cf-for-k8s

The open source deployment manifest for Cloud Foundry on Kubernetes
Apache License 2.0
300 stars 115 forks source link

Requests get routed to a stopped app if a route is attached #158

Open stoyanr opened 4 years ago

stoyanr commented 4 years ago

Summary

One of our components pushes 2 apps, both attached to the same route. One of them is never started (by design). However, the VirtualService generated for that route contains both applications and traffic is split 50 / 50. This leads to sporadic 503 errors when sending requests to that route.

Reproduction Steps

Push 2 apps without starting, map them to the same route, start only the first one. Then try requesting it. You will get 503 for some of the requests, e.g.:

$ curl -v https://internal-xsuaa.authentication.cf.dev5b.cp-k8s.shoot.canary.k8s-hana.ondemand.com/login
...
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
< HTTP/2 503
< content-length: 19
< content-type: text/plain
< date: Thu, 23 Apr 2020 13:29:53 GMT
< server: istio-envoy
<
* Connection #0 to host internal-xsuaa.authentication.cf.dev5b.cp-k8s.shoot.canary.k8s-hana.ondemand.com left intact
no healthy upstream* Closing connection 0

The VirtualService for the route looks like this:

$ k describe virtualservice vs-3573f4fc2f01caef34d152640066875f250ad36e2b84fb232796a3f8d9a4798a -n cf-workloads
Name:         vs-3573f4fc2f01caef34d152640066875f250ad36e2b84fb232796a3f8d9a4798a
Namespace:    cf-workloads
...
Spec:
  Gateways:
    istio-ingress
  Hosts:
    internal-xsuaa.authentication.cf.dev5b.cp-k8s.shoot.canary.k8s-hana.ondemand.com
  Http:
    Route:
      Destination:
        Host:  s-54c8ac95-b1e8-4ee6-8c82-e6868e57c87f
      Headers:
        Request:
          Set:
            CF - App - Id:              1ccd8cc0-5f07-423e-a40a-c0854d3205ea
            CF - App - Process - Type:  web
            CF - Organization - Id:     036a4deb-cf79-4ad4-bc5e-446ee5b92838
            CF - Space - Id:            2d280293-c1b5-46c9-85c6-7fc9e624e072
        Response:
      Weight:  50
      Destination:
        Host:  s-a244707f-f1d5-4cc1-ac3d-d7bb508f1aa8
      Headers:
        Request:
          Set:
            CF - App - Id:              f5f31886-3a8c-4db3-99eb-f06d48704ea3
            CF - App - Process - Type:  web
            CF - Organization - Id:     036a4deb-cf79-4ad4-bc5e-446ee5b92838
            CF - Space - Id:            2d280293-c1b5-46c9-85c6-7fc9e624e072
        Response:
      Weight:  50
Events:        <none>

Expected behavior

While an application is stopped, no requests should be routed to it, even if a route is attached. This is the behavior in BOSH CF and kubecf.

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/172478312

The labels on this github issue will be updated when the story is started.

christianang commented 4 years ago

Thanks for the report. This does seem like a problem. We will take a look and fix it.

cc @keshav-pivotal

davewalter commented 4 years ago

Marked this as a known issue for now. Please see https://www.pivotaltracker.com/story/show/172504850 for work to address this.