We recently added gateway loopback tests to Gold and Gold DR. However, The loopback routes were not being created in the standby cluster (ie: Gold DR) in test and production environments. It seems there was an issue with the gwa-scheduler (which retrieves Kong configurations from the dataplane) as well as the kube-api (which actually deploys/syncs the routes). It appeared that the scheduler was able to retrieve the routes from Kong, however the routes weren't actually being synced.
There was not any immediate signs of an issue (eg: log errors, pods crashing, etc) within the scheduler or kube-api. To help pinpoint the issue, we should add additional logging to these microservices in case this happens again in the future. Logging should indicate what is happening within each microservice.
Note: the sync issue was resolved after simply restarting the the scheduler pods within each environment (Gold DR test + prod)
We recently added gateway loopback tests to Gold and Gold DR. However, The loopback routes were not being created in the standby cluster (ie: Gold DR) in test and production environments. It seems there was an issue with the gwa-scheduler (which retrieves Kong configurations from the dataplane) as well as the kube-api (which actually deploys/syncs the routes). It appeared that the scheduler was able to retrieve the routes from Kong, however the routes weren't actually being synced.
There was not any immediate signs of an issue (eg: log errors, pods crashing, etc) within the scheduler or kube-api. To help pinpoint the issue, we should add additional logging to these microservices in case this happens again in the future. Logging should indicate what is happening within each microservice.
Note: the sync issue was resolved after simply restarting the the scheduler pods within each environment (Gold DR test + prod)