Closed rustyjux closed 1 month ago
Ready for review. Changes worked as expected in dev and are currently deployed there.
With the most recent changes in kube_api in dev (running in Gold which is in standby mode), I observed a route needing syncing being deleted and then re-inserted on the next scheduler run.
Detailed observations:
there was an existing route not-sticky-ias9df9-dev
which had the aps.route.session.cookie.enabled
tag (so actually was sticky). was previously synced to both clsuters
I removed the aps.route.session.cookie.enabled
tag on the route in the gw config
the update of the route in GoldDR is immediate and fine
the first time sync occurs, the route is deleted in Gold and is missing - I actually observed in Route in OCP the route being absent
[2024-10-18 18:24:32,249] DEBUG routes.verify_and_create_routes: insert batch: [{'name': 'wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'selectTag': 'ns.gw-41c0c', 'host': 'not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'sessionCookieEnabled': False, 'dataPlane': 'konghd-kong-proxy'}]
[2024-10-18 18:24:32,249] DEBUG routes.verify_and_create_routes: delete batch: [{'name': 'wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'selectTag': 'ns.gw-41c0c', 'host': 'not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'dataPlane': 'konghd-kong-proxy', 'sessionCookieEnabled': True}]
[2024-10-18 18:24:32,250] DEBUG routes.verify_and_create_routes: Creating 1 routes - tmp /tmp/sync/20241018182432-4d18599f5c
[2024-10-18 18:24:33,235] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] ['not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca'] No override applied {}
[2024-10-18 18:24:33,235] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 matched ssl cert data-api.tls
[2024-10-18 18:24:33,235] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca (ver.3542253514)
[2024-10-18 18:24:33,236] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Prepared 1 routes
[2024-10-18 18:24:33,946] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Applied 1 routes
[2024-10-18 18:24:33,946] DEBUG routes.verify_and_create_routes: Deleting 1 routes
[2024-10-18 18:24:34,848] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Deleted route wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca
[2024-10-18 18:24:34,848] INFO h11_impl. send: 10.97.8.1:43580 - "POST /namespaces/gw-41c0c/routes/sync HTTP/1.1" 200
on the next sync round, the route is recreated
[2024-10-18 18:40:47,239] DEBUG routes.verify_and_create_routes: insert batch: [{'name': 'wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'selectTag': 'ns.gw-41c0c', 'host': 'not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'sessionCookieEnabled': False, 'dataPlane': 'konghd-kong-proxy'}]
[2024-10-18 18:40:47,239] DEBUG routes.verify_and_create_routes: delete batch: []
[2024-10-18 18:40:47,240] DEBUG routes.verify_and_create_routes: Creating 1 routes - tmp /tmp/sync/20241018184047-2e8e88f2aa
[2024-10-18 18:40:48,236] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] ['not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca'] No override applied {}
[2024-10-18 18:40:48,236] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 matched ssl cert data-api.tls
[2024-10-18 18:40:48,236] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca (ver.)
[2024-10-18 18:40:48,237] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Prepared 1 routes
[2024-10-18 18:40:48,841] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Applied 1 routes
[2024-10-18 18:40:48,841] INFO h11_impl. send: 10.97.8.1:19364 - "POST /namespaces/gw-41c0c/routes/sync HTTP/1.1" 200
update gw config to restore the aps.route.session.cookie.enabled
tag
next sync cycle the route is deleted again
[2024-10-18 18:57:02,236] DEBUG routes.verify_and_create_routes: insert batch: [{'name': 'wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'selectTag': 'ns.gw-41c0c', 'host': 'not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'sessionCookieEnabled': True, 'dataPlane': 'konghd-kong-proxy'}]
[2024-10-18 18:57:02,236] DEBUG routes.verify_and_create_routes: delete batch: [{'name': 'wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'selectTag': 'ns.gw-41c0c', 'host': 'not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'dataPlane': 'konghd-kong-proxy', 'sessionCookieEnabled': False}]
[2024-10-18 18:57:02,236] DEBUG routes.verify_and_create_routes: Creating 1 routes - tmp /tmp/sync/20241018185702-23d00a2cbe
[2024-10-18 18:57:03,136] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 matched ssl cert data-api.tls
[2024-10-18 18:57:03,136] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca (ver.3544619441)
[2024-10-18 18:57:03,136] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Prepared 1 routes
[2024-10-18 18:57:03,935] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Applied 1 routes
[2024-10-18 18:57:03,935] DEBUG routes.verify_and_create_routes: Deleting 1 routes
[2024-10-18 18:57:04,833] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Deleted route wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca
[2024-10-18 18:57:04,833] INFO h11_impl. send: 10.97.8.1:62568 - "POST /namespaces/gw-41c0c/routes/sync HTTP/1.1" 200
So why is delete now happening after insert? Presumably this was not always the behaviour, though it does come second in routes and was always based on route name https://github.com/bcgov/gwa-api/blob/eaa8ea6d96317a4ba1fde618fcfe91903533c986/microservices/kubeApi/routers/routes.py#L217
I have set kube api back to dev
branch in versions to avoid creating more issues
Hmm... I think the reason is that for deletion, it really just needs to know whether the name
is no longer in the source_routes. The name
is derived from the selectTag and host
so this insert+delete issue would not have happened (dataPlane
doesn't change). When you added sessionCookieEnabled
then differences get added into the delete list incorrectly. To fix, I would just compare the name
for building the delete_batch.
Aha thanks for digging in there. That's a tricky side effect from changing in_list()
which is used in the comparison. I updated tests to reflect where deletes should and should not be happening.
Newest code works as expected when a sync update is needed in DR:
[2024-10-21 23:50:50,966] DEBUG routes.verify_and_create_routes: insert batch: [{'name': 'wild-ns-gw-41c0c-sticky-icky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'selectTag': 'ns.gw-41c0c', 'host': 'sticky-icky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'sessionCookieEnabled': False, 'dataPlane': 'konghd-kong-proxy'}]
[2024-10-21 23:50:50,966] DEBUG routes.verify_and_create_routes: delete batch: []
[2024-10-21 23:50:50,968] DEBUG routes.verify_and_create_routes: Creating 1 routes - tmp /tmp/sync/20241021235050-dfbc437d7c
[2024-10-21 23:50:51,956] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] ['sticky-icky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca'] No override applied {}
[2024-10-21 23:50:51,956] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 matched ssl cert data-api.tls
[2024-10-21 23:50:51,956] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 wild-ns-gw-41c0c-sticky-icky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca (ver.2834629484)
[2024-10-21 23:50:51,956] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Prepared 1 routes
[2024-10-21 23:50:52,585] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Applied 1 routes
[2024-10-21 23:50:52,585] INFO h11_impl. send: 10.95.10.1:14778 - "POST /namespaces/gw-41c0c/routes/sync HTTP/1.1" 200
Ugh probably should have squash merged. Could do a hard reset or we live with it
Description
Fixes APS-2743 - gwa scheduler is not using the route tag aps.route.session.cookie.enabled Includes sessionCookieEnabled when examining the existing routes in OCP on the cluster being synced
Types of changes
Checklist