cloudandheat / ch-k8s-lbaas

Flexible Loadbalancer-as-a-Service controller for Kubernetes
Apache License 2.0
9 stars 7 forks source link

SyncService race condition after restart #42

Open ibot3 opened 1 year ago

ibot3 commented 1 year ago

I encountered two problems that can occur during the initial service sync.

These are caused by the fact, that after restart, the controller does not have any state or knowledge of existing L3 ports. The controller only knows that a L3 port exists, if there is a service that references that port.

  1. If a new service is created between shutdown and start of the controller and is synced before (all) other services, the l3ports list is still empty, so no fitting port can be found for the new service. This causes a new L3 port being created, even if the service would fit on an existing L3 port (which is however not yet known by the controller).

  2. Imagine the controller is restarted and the following services are synced in the given order:

    • Existing service 1 (port-id: 3, L4-port: 1000)
    • New service (L4-port: 2000)
    • Existing service 2 (port-id: 3, L4-port: 2000)

    After the sync of Existing service 1, the L3 port 3 is known by the controller. When syncing New service, the controller would also assign it to L3-port 3, because the controller does not know yet, that L4-port 2000 is already used by Existing service 2. So the new service "steals" the port from Existing service 2 which must be relocated in this case.

Possible solutions: