[BUG] route-CRs don't get deleted

[phil9909]

Summary

kubectl get route.networking.cloudfoundry.org -n cf-workloads shows routes which have long been deleted from cf (cf routes does not show them).

Deployment Configuration

• cf-for-k8s version: https://github.com/akhinos/cf-for-k8s/commit/5eae6ad1dd60b2c8a81ae6c70e620f039de6c228
• cf-k8s-networking version: https://github.com/cloudfoundry/cf-k8s-networking/commit/2df0c84177c4421ae3e1662b7b1340451d984e4f
• Deploy command: Deployed using shalm
• Kubernetes CLI and API version: v1.16.9
• IaaS: Gardener on AWS

Reproduction Steps

1. create an org and space
2. push an app
3. delete app
4. delete route

or

1. create an org and space
2. push an app
3. delete org

Logs

I ran kubectl logs -n cf-system cfroutesync-76555f9659-p29tx cfroutesync

The only interesting thing I found in the log is this line

{"error":"route c74e8041-479a-403d-8a54-9e5d6fc7feb2 refers to missing space a7d509d2-cc7a-429e-88f1-68c6996b0ff4","level":"error","msg":"fetching","time":"2020-05-19T08:15:49Z"}

I still think it's unrelated, as the error appears only once, but I tried it multiple times.

Expected behavior

route.networking.cloudfoundry.org objects should be deleted once the cf-route is gone.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/172911044

The labels on this github issue will be updated when the story is started.

[jenspinney]

Hi @phil9909,

First of all, thank you for the clear bug report and deployment configuration breakdown. That really helps us to understand the issue quickly!

We have some general information that be may not be obvious and may help to contextualize this bug.

• We on the cf-k8s-networking team are in the process of re-architecting how app ingress works in cf-for-k8s.
  • cfroutesync + metacontroller will eventually be replaced by a new component called routecontroller.
• The cfroutesync architecture is what is currently deployed in the master branch of cf-for-k8s.
  • cfroutesync polls the cloud controller to observe when new CF routes appear, and then creates Services and (Istio) VirtualServices.
  • cfroutesync does nothing with Route CRs.
• The routecontroller archicture will look a little different. In that world, the cloud controller will create Route CRs in kubernetes. The routecontroller will react to these custom resources and create/update/delete Services and VirtualServices as necessary.
• At the moment, the cloud controller is producing these Route CRs, but nothing is reacting to those resources in cf-for-k8s yet (i.e., routecontroller doesn't yet exist in the master branch of cf-for-k8s).

With that said (and the understanding the Route CR stuff is very much still under development), the behavior you're seeing is not desired behavior. We weren't able to determine the commit of cloud controller that was deployed in the version of cf-for-k8s that you're using, but we have confirmed that cf-for-k8s master today does deploy a version of cloud controller that should implement the deletion of Route CRs (specifically, it includes this recently added line to implement Route CR deletion: https://github.com/cloudfoundry/cloud_controller_ng/blob/7bca62db869e213a07c2423f27decebfcdfe4c3c/app/models/runtime/route.rb#L251). Since this is such new functionality, it's possible that the story just wasn't done yet in the version of cf-for-k8s you're using.

If you upgrade to the latest cf-for-k8s, are you still seeing this behavior?

Thanks, Jen & @KauzClay

[phil9909]

If you upgrade to the latest cf-for-k8s, are you still seeing this behavior?

No, I don't see this behavior anymore. Thank you.