EIP can conflict between 2 clusters in same project due to tag naming

[deitch]

ccm uses EM EIP tags to determine which service an EIP is attached to (or intended for). That way it can stop, start, etc. and the affiliation will be appropriate. You even can pre-assign an EIP (which @c0dyhi11 wants to do with Anthos).

The logic when syncing services inside ccm is roughly (ignoring removal, etc.):

1. Find all services of type=LoadBalancer
2. Filter out any that already have Spec.LoadBalancerIP set
3. Create two tags:
   • usage=packet-ccm-auto
   • service=<service-identifier> (see below)
4. Find any EIP that have the above tags set
   • if found, use it
   • if not, create one, add the tags
5. Assign the EIP by setting Spec.LoadBalancerIP=<eip>

The <service-identifier> is created as follows (pseudo-code):

identifier = sha256( serviceNamespace + "/" +serviceName )

We use sha256 instead of just the clear text, because some customers did not want the names of their services exposed on the project configuration; it should remain internal to the cluster only.

The problem arises when you have 2 or more clusters in the same project. If both have a Service with the same name, they will conflict. It will try to assign the EIP to both of them, which is not what you want.

[deitch]

The solution is to have the EIP tagged not just by service and namespace, but by cluster as well.

I had thought about adding cluster identifier as part of the tag, i.e.:

identifier = sha256( clusterIdentiifier +"/" + serviceNamespace + "/" +serviceName )

but that makes that thing even more opaque, and makes it hard to separate resources (like EIPs) by cluster.

So I think we should have a separate cluster tag:

cluster=<clusterIdentifier>
service=<serviceIdentifier>

where the serviceIdentifier is the same as above.

As for clusterIdentifier, I do not know of any official unique cluster ID item, but it seems to be common practice to use the kube-system UID, e.g. on a kind cluster I just spun up:

apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2020-12-04T11:43:25Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:phase: {}
    manager: kube-apiserver
    operation: Update
    time: "2020-12-04T11:43:25Z"
  name: kube-system
  resourceVersion: "13"
  selfLink: /api/v1/namespaces/kube-system
  uid: fd23d60c-9bd5-4b00-8756-97a74cf232d8
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

So it would be fd23d60c-9bd5-4b00-8756-97a74cf232d8

[deitch]

@detiber is the above correct? Can we use that UID as a "unique cluster ID"?

[detiber]

@detiber is the above correct? Can we use that UID as a "unique cluster ID"?

Apologies for the delay. One thing to keep in mind about using a k8s resource UID is that if for some reason you need to restore a cluster from backup the UID for the resource will be different.

[deitch]

Fair enough. I think we can live with it for now.