envoyproxy / gateway

Manages Envoy Proxy as a Standalone or Kubernetes-based Application Gateway
https://gateway.envoyproxy.io
Apache License 2.0
1.55k stars 333 forks source link

Some status updates are lost #4336

Closed zhaohuabing closed 1 day ago

zhaohuabing commented 2 days ago

When upgrading Gateway API to v1.2.0, I discovered that the status updates were being discarded by the status updater, which caused the upgrade test to fail waiting for the HTTPRoute to be accepted.

Root cause: The status updater has been added as a Runnable in the controller manager, and started after the controller started watching resources. If a resource change comes before the status updater is started, the status will be discarded. This only happens when EG is just started.

Test log:

helpers.go:609: 2024-09-26T07:11:55.597278214Z: Route gateway-upgrade-infra/http-backend-eg-upgrade expected 1 Parents got 0
    helpers.go:609: 2024-09-26T07:11:56.597265108Z: Route gateway-upgrade-infra/http-backend-eg-upgrade expected 1 Parents got 0
    helpers.go:609: 2024-09-26T07:11:57.597740078Z: Route gateway-upgrade-infra/http-backend-eg-upgrade expected 1 Parents got 0
    helpers.go:609: 2024-09-26T07:11:58.596946119Z: Route gateway-upgrade-infra/http-backend-eg-upgrade expected 1 Parents got 0
    helpers.go:402: 
            Error Trace:    /home/ubuntu/go/pkg/mod/sigs.k8s.io/gateway-api@v1.2.0-rc1/conformance/utils/kubernetes/helpers.go:611
                                        /home/ubuntu/go/pkg/mod/sigs.k8s.io/gateway-api@v1.2.0-rc1/conformance/utils/kubernetes/helpers.go:367
                                        /home/ubuntu/go/pkg/mod/sigs.k8s.io/gateway-api@v1.2.0-rc1/conformance/utils/kubernetes/helpers.go:402
                                        /home/ubuntu/gateway/test/e2e/tests/eg_upgrade.go:120
            Error:          Received unexpected error:
                            error fetching HTTPRoute: client rate limiter Wait returned an error: context deadline exceeded
            Test:           TestEGUpgrade/EGUpgrade/Upgrade_from_an_older_eg_release_should_succeed
            Messages:       error waiting for HTTPRoute to have parents matching expectations

HTTPRoute, status was not updated.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  creationTimestamp: "2024-09-26T07:23:16Z"
  generation: 1
  name: http-backend-eg-upgrade
  namespace: gateway-upgrade-infra
  resourceVersion: "1433"
  uid: 6f995a87-a4d5-4400-96ba-dfb595944a4b
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: ha-gateway
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: infra-backend
      port: 8080
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /eg-upgrade

Failed tests:

probably related to https://github.com/envoyproxy/gateway/issues/4264