search

redhat-cop / egressip-ipam-operator

egressip-ipam-operator
Apache License 2.0
34 stars 23 forks source link

Nodes are not assigned egress IPs when using multiple CIDRs #69

Closed ctrought closed 3 years ago

ctrought commented 3 years ago

I am having issues with when using multiple CIDRs

OpenShifts docs say that

In the automatically assigned approach, an egress IP address range is assigned to a node. In the manually assigned approach, a list of one or more egress IP address is assigned to a node. https://docs.openshift.com/container-platform/4.6/networking/openshift_sdn/assigning-egress-ips.html

When I create one CIDR, the operator assigns one IP as expected to a given namespace. The IP address is made available on a node that matches the given selectors for the CIDR.

When I create two CIDR's, the operator assigns two IP addresses as expected to a given namespace. The IP addresses are not made available on any node, despite the operator logs appearing to correctly determine which nodes to assign the IP's to.

I am trying to figure out if this is an issue with OCP and the limitations of automatic vs manual, but in the readme for this operator and the fact it allows multiple CIDR's I am thinking it is expected to work? 

2021-05-28T18:10:42.009Z    DEBUG   controllers.EgressIPAM      {"CIDRs": ["10.36.80.240/29", "10.36.80.248/29"]}
2021-05-28T18:10:42.009Z    DEBUG   controllers.EgressIPAM      {"CIDRsByLabel": {"true":"10.36.80.248/29"}}
2021-05-28T18:10:42.009Z    DEBUG   controllers.EgressIPAM      {"reservedIPsByCIDR": {"10.36.80.240/29":[],"10.36.80.248/29":[]}}
2021-05-28T18:10:42.009Z    DEBUG   controllers.EgressIPAM      {"netCIDRByCIDR": {"10.36.80.240/29":{"IP":"10.36.80.240","Mask":"////+A=="},"10.36.80.248/29":{"IP":"10.36.80.248","Mask":"////+A=="}}}
2021-05-28T18:10:42.009Z    DEBUG   controllers.EgressIPAM      {"referringNamespaces": ["mynamespace"]}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM      {"initiallyAssignedNamespaces": ["mynamespace"]}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM      {"unAssignedNamespaces": []}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM      {"selectedNodes": ["node2.ocp.ca", "node1.ocp.ca", "node3.ocp.ca", "node4.ocp.ca"]}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM      {"selectedHostSubnets": ["node4.ocp.ca", "node1.ocp.ca", "node2.ocp.ca", "node3.ocp.ca"]}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM      {"selectedNodesByCIDR": {"10.36.80.248/29":["node2.ocp.ca","node1.ocp.ca","node3.ocp.ca"]}}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM      {"selectedHostSubnetByCIDR": {"10.36.80.248/29":["node2.ocp.ca","node1.ocp.ca","node3.ocp.ca"]}}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM      {"Used IPs By CIDR": {"10.36.80.240/29":[],"10.36.80.248/29":[]}}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM  currently assigned  {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241"],"10.36.80.248/29":["10.36.80.249"]}}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM  adding always excluded network IPs  {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247"],"10.36.80.248/29":["10.36.80.249","10.36.80.248","10.36.80.255"]}}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM  adding reserved IPs     {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247"],"10.36.80.248/29":["10.36.80.249","10.36.80.248","10.36.80.255"]}}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM  adding nodes IPs (if in the same CIDR)  {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247","10.36.80.240"],"10.36.80.248/29":["10.36.80.249","10.36.80.248","10.36.80.255"]}}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM  adding cloud infrastructure reserved IPs    {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247","10.36.80.240"],"10.36.80.248/29":["10.36.80.249","10.36.80.248","10.36.80.255"]}}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM  final   {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.247","10.36.80.241","10.36.80.240"],"10.36.80.248/29":["10.36.80.249","10.36.80.248","10.36.80.255"]}}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM  sorted reserved IPs     {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.240","10.36.80.241","10.36.80.247"],"10.36.80.248/29":["10.36.80.248","10.36.80.249","10.36.80.255"]}}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM      {"newlyAssignedNamespaces": []}
2021-05-28T18:10:42.010Z    DEBUG   controllers.EgressIPAM      {"finallyAssignedNamespaces": ["mynamespace"]}
2021-05-28T18:10:42.013Z    DEBUG   controllers.EgressIPAM      {"initiallyAssignedIPsByNode": {"node4.ocp.ca":[],"node1.ocp.ca":[],"node2.ocp.ca":[],"node3.ocp.ca":["10.36.80.241"]}}
2021-05-28T18:10:42.013Z    DEBUG   controllers.EgressIPAM      {"assignedIPsToNodesByCIDR: ": {"10.36.80.240/29":["10.36.80.241"],"10.36.80.248/29":[]}}
2021-05-28T18:10:42.013Z    DEBUG   controllers.EgressIPAM      {"assignedIPsToNamespaceByCIDR: ": {"10.36.80.240/29":["10.36.80.241"],"10.36.80.248/29":["10.36.80.249"]}}
2021-05-28T18:10:42.013Z    DEBUG   controllers.EgressIPAM      {"toBeAssignedToNodesIPsByCIDR: ": {"10.36.80.240/29":[],"10.36.80.248/29":["10.36.80.249"]}}
2021-05-28T18:10:42.013Z    DEBUG   controllers.EgressIPAM  new {"assignedIPsToNodesByCIDR: ": {"10.36.80.240/29":["10.36.80.241"],"10.36.80.248/29":[]}}
2021-05-28T18:10:42.013Z    DEBUG   controllers.EgressIPAM  new {"assignedIPsByNode: ": {"node3.ocp.ca":["10.36.80.241"]}}
2021-05-28T18:10:42.013Z    DEBUG   controllers.EgressIPAM      {"nodesByCIDR: ": {"10.36.80.248/29":["node2.ocp.ca","node1.ocp.ca","node3.ocp.ca"]}}
2021-05-28T18:10:42.013Z    DEBUG   controllers.EgressIPAM      {"nodesByNumberOfAssignedIPsByCIDR: ": {"10.36.80.248/29":{"0":["node2.ocp.ca","node1.ocp.ca"],"1":["node3.ocp.ca"]}}}
2021-05-28T18:10:42.013Z    DEBUG   controllers.EgressIPAM      {"nodesByNumberOfAssignedIPsByCIDR: ": {"10.36.80.248/29":{"0":["node2.ocp.ca","node1.ocp.ca"],"1":["node3.ocp.ca"]}}}
2021-05-28T18:10:42.013Z    DEBUG   controllers.EgressIPAM      {"minIPsPerNode: ": 0, "for cidr": "10.36.80.248/29"}
2021-05-28T18:10:42.013Z    INFO    controllers.EgressIPAM  assigning   {"IP": "10.36.80.249", "to node": "node2.ocp.ca"}
2021-05-28T18:10:42.013Z    DEBUG   controllers.EgressIPAM      {"finallyAssignedIPsByNode": {"node1.ocp.ca":[],"node2.ocp.ca":["10.36.80.249"],"node3.ocp.ca":["10.36.80.241"]}}

After logging into node2 and node3, there are no interfaces with this IP. And if checking the hostsubnet, we see the same.

$ oc get hostsubnet node2.ocp.ca
NAME                                       HOST                                       HOST IP        SUBNET          EGRESS CIDRS          EGRESS IPS
node2.ocp.ca                               node2.ocp.ca                               10.36.80.238   254.20.4.0/23   ["10.36.80.248/29"]

$ oc get hostsubnet node3.ocp.ca
NAME                                       HOST                                       HOST IP        SUBNET          EGRESS CIDRS          EGRESS IPS
node3.ocp.ca                               node3.ocp.ca                               10.36.80.239   254.20.4.0/23   ["10.36.80.240/29"]

If I remove CIDR2 from the egressip-ipam CR, an egress ip correctly gets added to a node from the only CIDR.

2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"CIDRs": ["10.36.80.240/29"]}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"CIDRsByLabel": {"true":"10.36.80.240/29"}}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"reservedIPsByCIDR": {"10.36.80.240/29":[]}}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"netCIDRByCIDR": {"10.36.80.240/29":{"IP":"10.36.80.240","Mask":"////+A=="}}}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"referringNamespaces": ["mynamespace"]}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"initiallyAssignedNamespaces": ["mynamespace"]}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"unAssignedNamespaces": []}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"selectedNodes": ["node2.ocp.ca", "node4.ocp.ca", "node3.ocp.ca", "node1.ocp.ca"]}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"selectedHostSubnets": ["node1.ocp.ca", "node2.ocp.ca", "node4.ocp.ca", "node3.ocp.ca"]}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"selectedNodesByCIDR": {"10.36.80.240/29":["node2.ocp.ca","node3.ocp.ca","node1.ocp.ca"]}}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"selectedHostSubnetByCIDR": {"10.36.80.240/29":["node2.ocp.ca","node3.ocp.ca","node1.ocp.ca"]}}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"Used IPs By CIDR": {"10.36.80.240/29":[]}}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM  currently assigned  {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241"]}}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM  adding always excluded network IPs  {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247"]}}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM  adding reserved IPs     {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247"]}}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM  adding nodes IPs (if in the same CIDR)  {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247","10.36.80.240"]}}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM  adding cloud infrastructure reserved IPs    {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247","10.36.80.240"]}}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM  final   {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.247","10.36.80.241","10.36.80.240"]}}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM  sorted reserved IPs     {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.240","10.36.80.241","10.36.80.247"]}}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"newlyAssignedNamespaces": []}
2021-05-28T18:57:38.283Z    DEBUG   controllers.EgressIPAM      {"finallyAssignedNamespaces": ["mynamespace"]}
2021-05-28T18:57:38.288Z    DEBUG   controllers.EgressIPAM      {"initiallyAssignedIPsByNode": {"node4.ocp.ca":[],"node1.ocp.ca":[],"node2.ocp.ca":[],"node3.ocp.ca":["10.36.80.241"]}}
2021-05-28T18:57:38.288Z    DEBUG   controllers.EgressIPAM      {"assignedIPsToNodesByCIDR: ": {"10.36.80.240/29":["10.36.80.241"]}}
2021-05-28T18:57:38.288Z    DEBUG   controllers.EgressIPAM      {"assignedIPsToNamespaceByCIDR: ": {"10.36.80.240/29":["10.36.80.241"]}}
2021-05-28T18:57:38.288Z    DEBUG   controllers.EgressIPAM      {"toBeAssignedToNodesIPsByCIDR: ": {"10.36.80.240/29":[]}}
2021-05-28T18:57:38.288Z    DEBUG   controllers.EgressIPAM  new {"assignedIPsToNodesByCIDR: ": {"10.36.80.240/29":["10.36.80.241"]}}
2021-05-28T18:57:38.288Z    DEBUG   controllers.EgressIPAM  new {"assignedIPsByNode: ": {"node3.ocp.ca":["10.36.80.241"]}}
2021-05-28T18:57:38.288Z    DEBUG   controllers.EgressIPAM      {"nodesByCIDR: ": {"10.36.80.240/29":["node2.ocp.ca","node3.ocp.ca","node1.ocp.ca"]}}
2021-05-28T18:57:38.288Z    DEBUG   controllers.EgressIPAM      {"nodesByNumberOfAssignedIPsByCIDR: ": {"10.36.80.240/29":{"0":["node2.ocp.ca","node1.ocp.ca"],"1":["node3.ocp.ca"]}}}
2021-05-28T18:57:38.288Z    DEBUG   controllers.EgressIPAM      {"finallyAssignedIPsByNode": {"node1.ocp.ca":[],"node2.ocp.ca":[],"node3.ocp.ca":["10.36.80.241"]}}
$ oc get hostsubnet node3.ocp.ca
NAME                                       HOST                                       HOST IP        SUBNET          EGRESS CIDRS          EGRESS IPS
node3.ocp.ca                               node3.ocp.ca                               10.36.80.239   254.20.4.0/23   ["10.36.80.240/29"]   ["10.36.80.241"]
raffaelespazzoli commented 3 years ago

can you share your EgressIPAM configuration and node labeling scheme?

ctrought commented 3 years ago

can you share your EgressIPAM configuration and node labeling scheme?

spec:
  cidrAssignments:
  - CIDR: 10.36.80.240/29
    labelValue: "true"
  - CIDR: 10.36.80.248/29
    labelValue: "false"
  nodeSelector:
    matchLabels:
      egress: "true"
  topologyLabel: virtual

Please ignore the logs from above, below are logs based on spec above. From the logs it looks like it assigns the ip's to the nodes, but as you can see from the hostsubnet no egress ip's are assigned to any nodes. The result is hung outbound network traffic from mynamespace. Thanks for taking a look

I0531 01:53:40.096596       1 request.go:655] Throttling request took 1.035760147s, request: GET:https://254.53.64.1:443/apis/authentication.k8s.io/v1?timeout=32s
2021-05-31T01:53:43.899Z    INFO    controller-runtime.metrics  metrics server is starting to listen    {"addr": ":8080"}
2021-05-31T01:53:48.761Z    INFO    setup   starting manager
I0531 01:53:48.761325       1 leaderelection.go:243] attempting to acquire leader lease egressip-ipam-operator/9ae943bf.redhat.io...
2021-05-31T01:53:48.761Z    INFO    controller-runtime.manager  starting metrics server {"path": "/metrics"}
I0531 01:54:06.175996       1 leaderelection.go:253] successfully acquired lease egressip-ipam-operator/9ae943bf.redhat.io
2021-05-31T01:54:06.176Z    DEBUG   controller-runtime.manager.events   Normal  {"object": {"kind":"ConfigMap","namespace":"egressip-ipam-operator","name":"9ae943bf.redhat.io","uid":"f5c1c2cb-4fdc-451a-87b5-f45899cba1d6","apiVersion":"v1","resourceVersion":"117538703"}, "reason": "LeaderElection", "message": "egressip-ipam-operator-controller-manager-69b469cc9f-zb7tc_514a5799-cfc4-4877-b6ac-5416c39ae3b8 became leader"}
2021-05-31T01:54:06.176Z    INFO    controller-runtime.manager.controller.namespace Starting EventSource    {"reconciler group": "", "reconciler kind": "Namespace", "source": "kind source: /, Kind="}
2021-05-31T01:54:06.176Z    INFO    controller-runtime.manager.controller.egressipam    Starting EventSource    {"reconciler group": "redhatcop.redhat.io", "reconciler kind": "EgressIPAM", "source": "kind source: /, Kind="}
2021-05-31T01:54:06.276Z    INFO    controller-runtime.manager.controller.namespace Starting Controller {"reconciler group": "", "reconciler kind": "Namespace"}
2021-05-31T01:54:06.276Z    INFO    controller-runtime.manager.controller.namespace Starting workers    {"reconciler group": "", "reconciler kind": "Namespace", "worker count": 1}
2021-05-31T01:54:06.276Z    INFO    controller-runtime.manager.controller.egressipam    Starting EventSource    {"reconciler group": "redhatcop.redhat.io", "reconciler kind": "EgressIPAM", "source": "kind source: /, Kind=Node"}
2021-05-31T01:54:06.377Z    INFO    controller-runtime.manager.controller.egressipam    Starting EventSource    {"reconciler group": "redhatcop.redhat.io", "reconciler kind": "EgressIPAM", "source": "kind source: /, Kind=HostSubnet"}
2021-05-31T01:54:06.477Z    INFO    controller-runtime.manager.controller.egressipam    Starting EventSource    {"reconciler group": "redhatcop.redhat.io", "reconciler kind": "EgressIPAM", "source": "kind source: /, Kind=Namespace"}
2021-05-31T01:54:06.477Z    INFO    controller-runtime.manager.controller.egressipam    Starting EventSource    {"reconciler group": "redhatcop.redhat.io", "reconciler kind": "EgressIPAM", "source": "kind source: /, Kind=NetNamespace"}
2021-05-31T01:54:06.578Z    INFO    controller-runtime.manager.controller.egressipam    Starting Controller {"reconciler group": "redhatcop.redhat.io", "reconciler kind": "EgressIPAM"}
2021-05-31T01:54:06.578Z    INFO    controller-runtime.manager.controller.egressipam    Starting workers    {"reconciler group": "redhatcop.redhat.io", "reconciler kind": "EgressIPAM", "worker count": 1}
2021-05-31T01:54:06.578Z    DEBUG   controllers.EgressIPAM      {"CIDRs": ["10.36.80.240/29", "10.36.80.248/29"]}
2021-05-31T01:54:06.578Z    DEBUG   controllers.EgressIPAM      {"CIDRsByLabel": {"false":"10.36.80.248/29","true":"10.36.80.240/29"}}
2021-05-31T01:54:06.578Z    DEBUG   controllers.EgressIPAM      {"reservedIPsByCIDR": {"10.36.80.240/29":[],"10.36.80.248/29":[]}}
2021-05-31T01:54:06.578Z    DEBUG   controllers.EgressIPAM      {"netCIDRByCIDR": {"10.36.80.240/29":{"IP":"10.36.80.240","Mask":"////+A=="},"10.36.80.248/29":{"IP":"10.36.80.248","Mask":"////+A=="}}}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM      {"referringNamespaces": ["mynamespace"]}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM      {"initiallyAssignedNamespaces": ["mynamespace"]}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM      {"unAssignedNamespaces": []}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM      {"selectedNodes": ["node1.ocp.ca", "node3.ocp.ca", "node4.ocp.ca", "node2.ocp.ca"]}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM      {"selectedHostSubnets": ["node4.ocp.ca", "node2.ocp.ca", "node1.ocp.ca", "node3.ocp.ca"]}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM      {"selectedNodesByCIDR": {"10.36.80.240/29":["node2.ocp.ca","node1.ocp.ca","node3.ocp.ca"],"10.36.80.248/29":["node4.ocp.ca"]}}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM      {"selectedHostSubnetByCIDR": {"10.36.80.240/29":["node2.ocp.ca","node1.ocp.ca","node3.ocp.ca"],"10.36.80.248/29":["node4.ocp.ca"]}}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM      {"Used IPs By CIDR": {"10.36.80.240/29":[],"10.36.80.248/29":[]}}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM  currently assigned  {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241"],"10.36.80.248/29":["10.36.80.249"]}}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM  adding always excluded network IPs  {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247"],"10.36.80.248/29":["10.36.80.249","10.36.80.248","10.36.80.255"]}}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM  adding reserved IPs     {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247"],"10.36.80.248/29":["10.36.80.249","10.36.80.248","10.36.80.255"]}}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM  adding nodes IPs (if in the same CIDR)  {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247","10.36.80.240"],"10.36.80.248/29":["10.36.80.249","10.36.80.248","10.36.80.255"]}}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM  adding cloud infrastructure reserved IPs    {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247","10.36.80.240"],"10.36.80.248/29":["10.36.80.249","10.36.80.248","10.36.80.255"]}}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM  final   {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247"],"10.36.80.248/29":["10.36.80.255","10.36.80.249","10.36.80.248"]}}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM  sorted reserved IPs     {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.240","10.36.80.241","10.36.80.247"],"10.36.80.248/29":["10.36.80.248","10.36.80.249","10.36.80.255"]}}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM      {"newlyAssignedNamespaces": []}
2021-05-31T01:54:06.579Z    DEBUG   controllers.EgressIPAM      {"finallyAssignedNamespaces": ["mynamespace"]}
2021-05-31T01:54:06.584Z    DEBUG   controllers.EgressIPAM      {"initiallyAssignedIPsByNode": {"node4.ocp.ca":[],"node1.ocp.ca":[],"node2.ocp.ca":[],"node3.ocp.ca":[]}}
2021-05-31T01:54:06.584Z    DEBUG   controllers.EgressIPAM      {"assignedIPsToNodesByCIDR: ": {"10.36.80.240/29":[],"10.36.80.248/29":[]}}
2021-05-31T01:54:06.584Z    DEBUG   controllers.EgressIPAM      {"assignedIPsToNamespaceByCIDR: ": {"10.36.80.240/29":["10.36.80.241"],"10.36.80.248/29":["10.36.80.249"]}}
2021-05-31T01:54:06.584Z    DEBUG   controllers.EgressIPAM      {"toBeAssignedToNodesIPsByCIDR: ": {"10.36.80.240/29":["10.36.80.241"],"10.36.80.248/29":["10.36.80.249"]}}
2021-05-31T01:54:06.584Z    DEBUG   controllers.EgressIPAM  new {"assignedIPsToNodesByCIDR: ": {"10.36.80.240/29":[],"10.36.80.248/29":[]}}
2021-05-31T01:54:06.584Z    DEBUG   controllers.EgressIPAM  new {"assignedIPsByNode: ": {}}
2021-05-31T01:54:06.584Z    DEBUG   controllers.EgressIPAM      {"nodesByCIDR: ": {"10.36.80.240/29":["node2.ocp.ca","node1.ocp.ca","node3.ocp.ca"],"10.36.80.248/29":["node4.ocp.ca"]}}
2021-05-31T01:54:06.584Z    DEBUG   controllers.EgressIPAM      {"nodesByNumberOfAssignedIPsByCIDR: ": {"10.36.80.240/29":{"0":["node2.ocp.ca","node1.ocp.ca","node3.ocp.ca"]},"10.36.80.248/29":{"0":["node4.ocp.ca"]}}}
2021-05-31T01:54:06.584Z    DEBUG   controllers.EgressIPAM      {"nodesByNumberOfAssignedIPsByCIDR: ": {"10.36.80.240/29":{"0":["node2.ocp.ca","node1.ocp.ca","node3.ocp.ca"]},"10.36.80.248/29":{"0":["node4.ocp.ca"]}}}
2021-05-31T01:54:06.584Z    DEBUG   controllers.EgressIPAM      {"minIPsPerNode: ": 0, "for cidr": "10.36.80.240/29"}
2021-05-31T01:54:06.584Z    INFO    controllers.EgressIPAM  assigning   {"IP": "10.36.80.241", "to node": "node2.ocp.ca"}
2021-05-31T01:54:06.584Z    DEBUG   controllers.EgressIPAM      {"nodesByNumberOfAssignedIPsByCIDR: ": {"10.36.80.240/29":{"0":["node1.ocp.ca","node3.ocp.ca"],"1":["node2.ocp.ca"]},"10.36.80.248/29":{"0":["node4.ocp.ca"]}}}
2021-05-31T01:54:06.584Z    DEBUG   controllers.EgressIPAM      {"minIPsPerNode: ": 0, "for cidr": "10.36.80.248/29"}
2021-05-31T01:54:06.584Z    INFO    controllers.EgressIPAM  assigning   {"IP": "10.36.80.249", "to node": "node4.ocp.ca"}
2021-05-31T01:54:06.584Z    DEBUG   controllers.EgressIPAM      {"finallyAssignedIPsByNode": {"node4.ocp.ca":["10.36.80.249"],"node1.ocp.ca":[],"node2.ocp.ca":["10.36.80.241"],"node3.ocp.ca":[]}}
2021-05-31T01:54:06.594Z    DEBUG   controllers.EgressIPAM      {"CIDRs": ["10.36.80.240/29", "10.36.80.248/29"]}
2021-05-31T01:54:06.594Z    DEBUG   controllers.EgressIPAM      {"CIDRsByLabel": {"false":"10.36.80.248/29","true":"10.36.80.240/29"}}
2021-05-31T01:54:06.594Z    DEBUG   controllers.EgressIPAM      {"reservedIPsByCIDR": {"10.36.80.240/29":[],"10.36.80.248/29":[]}}
2021-05-31T01:54:06.594Z    DEBUG   controllers.EgressIPAM      {"netCIDRByCIDR": {"10.36.80.240/29":{"IP":"10.36.80.240","Mask":"////+A=="},"10.36.80.248/29":{"IP":"10.36.80.248","Mask":"////+A=="}}}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM      {"referringNamespaces": ["mynamespace"]}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM      {"initiallyAssignedNamespaces": ["mynamespace"]}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM      {"unAssignedNamespaces": []}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM      {"selectedNodes": ["node2.ocp.ca", "node3.ocp.ca", "node1.ocp.ca", "node4.ocp.ca"]}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM      {"selectedHostSubnets": ["node2.ocp.ca", "node1.ocp.ca", "node4.ocp.ca", "node3.ocp.ca"]}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM      {"selectedNodesByCIDR": {"10.36.80.240/29":["node1.ocp.ca","node2.ocp.ca","node3.ocp.ca"],"10.36.80.248/29":["node4.ocp.ca"]}}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM      {"selectedHostSubnetByCIDR": {"10.36.80.240/29":["node1.ocp.ca","node2.ocp.ca","node3.ocp.ca"],"10.36.80.248/29":["node4.ocp.ca"]}}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM      {"Used IPs By CIDR": {"10.36.80.240/29":[],"10.36.80.248/29":[]}}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM  currently assigned  {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241"],"10.36.80.248/29":["10.36.80.249"]}}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM  adding always excluded network IPs  {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247"],"10.36.80.248/29":["10.36.80.249","10.36.80.248","10.36.80.255"]}}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM  adding reserved IPs     {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247"],"10.36.80.248/29":["10.36.80.249","10.36.80.248","10.36.80.255"]}}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM  adding nodes IPs (if in the same CIDR)  {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247","10.36.80.240"],"10.36.80.248/29":["10.36.80.249","10.36.80.248","10.36.80.255"]}}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM  adding cloud infrastructure reserved IPs    {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247","10.36.80.240"],"10.36.80.248/29":["10.36.80.249","10.36.80.248","10.36.80.255"]}}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM  final   {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.241","10.36.80.240","10.36.80.247"],"10.36.80.248/29":["10.36.80.248","10.36.80.255","10.36.80.249"]}}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM  sorted reserved IPs     {"IPs by CIDR": {"10.36.80.240/29":["10.36.80.240","10.36.80.241","10.36.80.247"],"10.36.80.248/29":["10.36.80.248","10.36.80.249","10.36.80.255"]}}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM      {"newlyAssignedNamespaces": []}
2021-05-31T01:54:06.595Z    DEBUG   controllers.EgressIPAM      {"finallyAssignedNamespaces": ["mynamespace"]}
2021-05-31T01:54:06.600Z    DEBUG   controllers.EgressIPAM      {"initiallyAssignedIPsByNode": {"node4.ocp.ca":[],"node1.ocp.ca":[],"node2.ocp.ca":[],"node3.ocp.ca":[]}}
2021-05-31T01:54:06.600Z    DEBUG   controllers.EgressIPAM      {"assignedIPsToNodesByCIDR: ": {"10.36.80.240/29":[],"10.36.80.248/29":[]}}
2021-05-31T01:54:06.600Z    DEBUG   controllers.EgressIPAM      {"assignedIPsToNamespaceByCIDR: ": {"10.36.80.240/29":["10.36.80.241"],"10.36.80.248/29":["10.36.80.249"]}}
2021-05-31T01:54:06.600Z    DEBUG   controllers.EgressIPAM      {"toBeAssignedToNodesIPsByCIDR: ": {"10.36.80.240/29":["10.36.80.241"],"10.36.80.248/29":["10.36.80.249"]}}
2021-05-31T01:54:06.600Z    DEBUG   controllers.EgressIPAM  new {"assignedIPsToNodesByCIDR: ": {"10.36.80.240/29":[],"10.36.80.248/29":[]}}
2021-05-31T01:54:06.600Z    DEBUG   controllers.EgressIPAM  new {"assignedIPsByNode: ": {}}
2021-05-31T01:54:06.600Z    DEBUG   controllers.EgressIPAM      {"nodesByCIDR: ": {"10.36.80.240/29":["node1.ocp.ca","node2.ocp.ca","node3.ocp.ca"],"10.36.80.248/29":["node4.ocp.ca"]}}
2021-05-31T01:54:06.600Z    DEBUG   controllers.EgressIPAM      {"nodesByNumberOfAssignedIPsByCIDR: ": {"10.36.80.240/29":{"0":["node1.ocp.ca","node2.ocp.ca","node3.ocp.ca"]},"10.36.80.248/29":{"0":["node4.ocp.ca"]}}}
2021-05-31T01:54:06.600Z    DEBUG   controllers.EgressIPAM      {"nodesByNumberOfAssignedIPsByCIDR: ": {"10.36.80.240/29":{"0":["node1.ocp.ca","node2.ocp.ca","node3.ocp.ca"]},"10.36.80.248/29":{"0":["node4.ocp.ca"]}}}
2021-05-31T01:54:06.600Z    DEBUG   controllers.EgressIPAM      {"minIPsPerNode: ": 0, "for cidr": "10.36.80.240/29"}
2021-05-31T01:54:06.600Z    INFO    controllers.EgressIPAM  assigning   {"IP": "10.36.80.241", "to node": "node1.ocp.ca"}
2021-05-31T01:54:06.600Z    DEBUG   controllers.EgressIPAM      {"nodesByNumberOfAssignedIPsByCIDR: ": {"10.36.80.240/29":{"0":["node2.ocp.ca","node3.ocp.ca"],"1":["node1.ocp.ca"]},"10.36.80.248/29":{"0":["node4.ocp.ca"]}}}
2021-05-31T01:54:06.600Z    DEBUG   controllers.EgressIPAM      {"minIPsPerNode: ": 0, "for cidr": "10.36.80.248/29"}
2021-05-31T01:54:06.600Z    INFO    controllers.EgressIPAM  assigning   {"IP": "10.36.80.249", "to node": "node4.ocp.ca"}
2021-05-31T01:54:06.600Z    DEBUG   controllers.EgressIPAM      {"finallyAssignedIPsByNode": {"node4.ocp.ca":["10.36.80.249"],"node1.ocp.ca":["10.36.80.241"],"node2.ocp.ca":[],"node3.ocp.ca":[]}}
$ oc get nodes -l virtual=true
NAME                    STATUS   ROLES     AGE   VERSION
node1.ocp.ca            Ready    virtual   22d   v1.20.0+df9c838
node2.ocp.ca            Ready    virtual   79d   v1.20.0+df9c838
node3.ocp.ca            Ready    virtual   79d   v1.20.0+df9c838

$ oc get nodes -l virtual=false
NAME                    STATUS   ROLES    AGE   VERSION
node4.ocp.ca            Ready    worker   22d   v1.20.0+df9c838

$ oc get nodes -l egress=true
NAME                    STATUS   ROLES     AGE   VERSION
node1.ocp.ca            Ready    virtual   22d   v1.20.0+df9c838
node2.ocp.ca            Ready    virtual   79d   v1.20.0+df9c838
node3.ocp.ca            Ready    virtual   79d   v1.20.0+df9c838
node4.ocp.ca            Ready    worker    22d   v1.20.0+df9c838

$ oc get hostsubnet
NAME                    HOST                    HOST IP        SUBNET          EGRESS CIDRS          EGRESS IPS
infra1.ocp.ca           infra1.ocp.ca           10.36.80.234   254.20.2.0/23
infra2.ocp.ca           infra2.ocp.ca           10.36.80.235   254.21.2.0/23
infra3.ocp.ca           infra3.ocp.ca           10.36.80.236   254.23.0.0/23
master1.ocp.ca          master1.ocp.ca          10.36.80.231   254.20.0.0/23
master2.ocp.ca          master2.ocp.ca          10.36.80.232   254.21.0.0/23
master3.ocp.ca          master3.ocp.ca          10.36.80.233   254.22.0.0/23
node1.ocp.ca            node1.ocp.ca            10.36.80.237   254.22.4.0/23   ["10.36.80.240/29"]
node2.ocp.ca            node2.ocp.ca            10.36.80.238   254.20.4.0/23   ["10.36.80.240/29"]
node3.ocp.ca            node3.ocp.ca            10.36.80.239   254.23.2.0/23   ["10.36.80.240/29"]
node4.ocp.ca            node4.ocp.ca            10.36.80.240   254.21.4.0/23   ["10.36.80.248/29"]
raffaelespazzoli commented 3 years ago

what infrastructure are you running on? can you share the result of oc get infrastructure cluster -o yaml it looks like someone or something has assigned CIDRs rather than EgressIPs to the nodes. This should happen when you run on something that is not AWS or Azure. The assignments seem correct, did you do it or did the operator do it? can you also share oc get netnamespace you should see the assigned IPs there. once you see the assigned IPs there, that is all you need to start working.

ctrought commented 3 years ago

what infrastructure are you running on? can you share the result of oc get infrastructure cluster -o yaml

On prem (bare metal)

spec:
  cloudConfig:
    name: ""
  platformSpec:
    type: None
status:
  apiServerInternalURI: https://api-int.ocp-lab.ca:6443
  apiServerURL: https://api.ocp-lab.ca:6443
  etcdDiscoveryDomain: ocp-lab
  infrastructureName: ocp-lab
  platform: None
  platformStatus:
    type: None

it looks like someone or something has assigned CIDRs rather than EgressIPs to the nodes. This should happen when you run on something that is not AWS or Azure. The assignments seem correct, did you do it or did the operator do it? can you also share

My understanding was I should see an egress IP assigned to the node (in addition to CIDR) which is the behaviour seen when using a single CIDR (see output at end). The egress CIDR's seen from oc get hostsubnet are only assigned when using this operator, if I remove the CR they are all removed from the nodes.

oc get netnamespace you should see the assigned IPs there. once you see the assigned IPs there, that is all you need to start working.

$ oc get netnamespaces mynamespace   
NAME          NETID     EGRESS IPS
mynamespace   7141923   ["10.36.80.241","10.36.80.249"]

IP's are correctly assigned to the namespace. No outbound network traffic works though. If I remove 1 of the CIDR's from the operator CR then outbound network traffic starts flowing again through the one CIDR/egress IP, and the egress IP for the namespace is seen on the hostsubnet.

No CIDRs (remove egressipam CR)

$ oc get netnamespaces mynamespace   
NAME          NETID     EGRESS IPS
mynamespace   7141923

$ oc get hostsubnet
NAME                    HOST                    HOST IP        SUBNET          EGRESS CIDRS          EGRESS IPS
infra1.ocp.ca           infra1.ocp.ca           10.36.80.234   254.20.2.0/23
infra2.ocp.ca           infra2.ocp.ca           10.36.80.235   254.21.2.0/23
infra3.ocp.ca           infra3.ocp.ca           10.36.80.236   254.23.0.0/23
master1.ocp.ca          master1.ocp.ca          10.36.80.231   254.20.0.0/23
master2.ocp.ca          master2.ocp.ca          10.36.80.232   254.21.0.0/23
master3.ocp.ca          master3.ocp.ca          10.36.80.233   254.22.0.0/23
node1.ocp.ca            node1.ocp.ca            10.36.80.237   254.22.4.0/23
node2.ocp.ca            node2.ocp.ca            10.36.80.238   254.20.4.0/23
node3.ocp.ca            node3.ocp.ca            10.36.80.239   254.23.2.0/23
node4.ocp.ca            node4.ocp.ca            10.36.80.240   254.21.4.0/23

Single CIDR

$ oc get netnamespaces mynamespace   
NAME          NETID     EGRESS IPS
mynamespace   7141923   ["10.36.80.249"]

$ oc get hostsubnet
NAME                    HOST                    HOST IP        SUBNET          EGRESS CIDRS          EGRESS IPS
infra1.ocp.ca           infra1.ocp.ca           10.36.80.234   254.20.2.0/23
infra2.ocp.ca           infra2.ocp.ca           10.36.80.235   254.21.2.0/23
infra3.ocp.ca           infra3.ocp.ca           10.36.80.236   254.23.0.0/23
master1.ocp.ca          master1.ocp.ca          10.36.80.231   254.20.0.0/23
master2.ocp.ca          master2.ocp.ca          10.36.80.232   254.21.0.0/23
master3.ocp.ca          master3.ocp.ca          10.36.80.233   254.22.0.0/23
node1.ocp.ca            node1.ocp.ca            10.36.80.237   254.22.4.0/23   
node2.ocp.ca            node2.ocp.ca            10.36.80.238   254.20.4.0/23   
node3.ocp.ca            node3.ocp.ca            10.36.80.239   254.23.2.0/23  
node4.ocp.ca            node4.ocp.ca            10.36.80.240   254.21.4.0/23   ["10.36.80.248/29"]   ["10.36.80.249"]

Two CIDR's

$ oc get netnamespaces mynamespace   
NAME          NETID     EGRESS IPS
mynamespace   7141923   ["10.36.80.241","10.36.80.249"]

$ oc get hostsubnet
NAME                    HOST                    HOST IP        SUBNET          EGRESS CIDRS          EGRESS IPS
infra1.ocp.ca           infra1.ocp.ca           10.36.80.234   254.20.2.0/23
infra2.ocp.ca           infra2.ocp.ca           10.36.80.235   254.21.2.0/23
infra3.ocp.ca           infra3.ocp.ca           10.36.80.236   254.23.0.0/23
master1.ocp.ca          master1.ocp.ca          10.36.80.231   254.20.0.0/23
master2.ocp.ca          master2.ocp.ca          10.36.80.232   254.21.0.0/23
master3.ocp.ca          master3.ocp.ca          10.36.80.233   254.22.0.0/23
node1.ocp.ca            node1.ocp.ca            10.36.80.237   254.22.4.0/23   ["10.36.80.240/29"]
node2.ocp.ca            node2.ocp.ca            10.36.80.238   254.20.4.0/23   ["10.36.80.240/29"]
node3.ocp.ca            node3.ocp.ca            10.36.80.239   254.23.2.0/23   ["10.36.80.240/29"]
node4.ocp.ca            node4.ocp.ca            10.36.80.240   254.21.4.0/23   ["10.36.80.248/29"]

From OCP docs. So I guess this is the behaviour I am seeing (no host hosts the egress IP, traffic dropped). I am not clear why it's not hosting the egress IP in multi-cidr mode but single cidr works.

Namespaces that request an egress IP address are matched with nodes that can host those egress IP addresses, and then the egress IP addresses are assigned to those nodes. If the egressIPs parameter is set on a NetNamespace object, but no node hosts that egress IP address, then egress traffic from the namespace will be dropped.

raffaelespazzoli commented 3 years ago

ok, thanks. For your information, when using baremetal this operator does not assign egressIPs to nodes, OpenShift should do that. In this case this operator assigns EgressIPs to namespaces and CIDRs to nodes. If you create the CR without any namespace using the egressIP, you should see the CIDRs being assigned to the nodes. The logs might be misleading as you see all the reasoning the operator does about where it would put the EgressIPs, but then it actually does not do anything with it. That piece of the algorithm is only used when in the cloud. So, as far as I can tell, the operator does what it expected, and you might be seeing an OCP issue. Can you try to reproduce it by creating the configuration manually? remove the operator, assign the CIDRs to the nodes, then assign two IPs to a namespace, let's see what happens.

ctrought commented 3 years ago

ok, thanks. For your information, when using baremetal this operator does not assign egressIPs to nodes, OpenShift should do that. In this case this operator assigns EgressIPs to namespaces and CIDRs to nodes. If you create the CR without any namespace using the egressIP, you should see the CIDRs being assigned to the nodes. The logs might be misleading as you see all the reasoning the operator does about where it would put the EgressIPs, but then it actually does not do anything with it. That piece of the algorithm is only used when in the cloud. So, as far as I can tell, the operator does what it expected, and you might be seeing an OCP issue. Can you try to reproduce it by creating the configuration manually? remove the operator, assign the CIDRs to the nodes, then assign two IPs to a namespace, let's see what happens.

Thank you @raffaelespazzoli. I will try manually and update you, if needed I'll open a case with RH.

raffaelespazzoli commented 3 years ago

may I close this?

snorwin commented 3 years ago

@raffaelespazzoli this is still an issue. I was able to reproduce it using OpenShift 4.6.16 with OpenShift SDN on baremetal.

I was checking the documentation (https://docs.openshift.com/container-platform/4.7/networking/openshift_sdn/assigning-egress-ips.html) and only a single egress IP address per namespace is supported when using the automatic assignment mode where a CIDR is assigned to the node. Otherwise if a namespace has multiple egress IPs, each IP need to be assigned manually to the node. Manually I was able to verify those statements of the documentation.

I think we should enhance the following if-else condition: https://github.com/redhat-cop/egressip-ipam-operator/blob/e8f46d6e150d38cf68fbdcd816b8b6f82c09d4f7/controllers/egressipam/egressipam_controller.go#L202

raffaelespazzoli commented 3 years ago

So you are creating two cidr, but your nodes actually belong to the same subnet which you are artificially subdividing. then when it's time assign cidrs to nodes OCP does not know what to do because for OCP all nodes are in the same network... Is that a good summary of the problem? Why do you need two CIDRs when you have a single network?

snorwin commented 3 years ago

We run an OpenShift cluster spread over two location. In each location there is a subnet used for EgressIP with special router nodes in that subnet. In case of site failure it could be that only one master node survived which would cause etcd to be in read only mode. Therefore it is important that each namespace already have two EgressIPs assigned, one of each site's subnet.

The issue we encounter is that if you use CIDR in the HostSubnets in combination with multiple IPs defined in the NetNamspace it is not working (as described in the documentation). In order to be able to use multiple EgressIPs in one namespace the IPs need be assigned manually to the node.