search

cosmonic-labs / netreap

A Cilium controller implementation for Nomad
https://netreap.io
Apache License 2.0
129 stars 8 forks source link

Refactor endpoint handling and reconcilliation #21

Closed deverton-godaddy closed 1 year ago

deverton-godaddy commented 1 year ago

Feature or Problem

With the release of Nomad 1.6 it's possible to get the network address of the allocation from Nomad. The change to enable this is only in the client library and does not require updating the Nomad server. The IP was sent back by older Nomad versions, it just wasn't available in the client.

This enables refactoring the endpoint reconcilliation to make use of the IP address to identify the endpoint within Cilium. There is no longer a dependency on Consul for policy reconcilliaton. Additionally, endpoints are now labelled with the task group and task information as services can be created at those levels.

On startup of netreap the reconciliation process is as follows:

  1. List all the endpoints in Cilium
  2. Get the container ID associated with the endpoint. As part of the CNI call from Nomad, it will pass the allocation ID as the container ID to Cilium.
  3. Use the container ID to look up the allocation in Nomad.
  4. Label the endpoints as needed.

There's no need to delete endpoints (as far as I can tell) as Nomad should be deleting the endpoints as part of its integration with the CNI plugin.

For event processing netreap will now monitor the Allocation topic as the AllocationUpdated event from that topic contains the network details associated with the allocation. The process is now:

  1. Receive an AllocationUpdated event.
  2. If it does not have an IP address associated (and tasks without network configuration won't) ignore it.
  3. Use the IP address to look up the endpoint in the local Cilium agent. You can query endpoints by IP address and Cilium will only return details for endpoints that re local to the node. Though we could also just query by container ID (i.e. allocation ID).
  4. If the allocation event doesn't include the Job data, fetch it and then apply the labels combing information from both.

This should be fairly scalable as the number of calls to the Nomad API will be one per-endpoint allocation.

One other change is patch the endpoint with an endpoint change request event. This triggers Cilium to remove the reserved:init label from the endpoints as it considers the endpoints labelled properly at that point. This change is based on how the Cilium Docker network plugin works.

Note that it is no longer possible to filter out services by tags. I'm not sure that feature makes sense with the refactored code as you would always want to apply labels to Cilium endpoints.

Related Issues

Potentially fixes https://github.com/cosmonic/netreap/issues/20 and https://github.com/cosmonic/netreap/issues/9

netlify[bot] commented 1 year ago

Deploy Preview for netreap canceled.

Name Link
Latest commit a21fe9bd5953ccab474be375939711032ef788a5
Latest deploy log https://app.netlify.com/sites/netreap/deploys/64c97bee3387290008ed29f2