How do we support static IP address allocation in CNI?

[asridharan]

Currently the spec does not allow orchestration engines to request allocation of a specific IP address from the plugins. Are there any thoughts/proposals to support static IP addresses in CNI?

[squeed]

This would be up to the individual IPAM plugin.

Currently, the host-local plugin supports this through the ip=xxx CNI-arg. Check out https://github.com/containernetworking/cni/blob/master/Documentation/host-local.md

Am Oct 8, 2016 um 2:29 AM schrieb asridharan notifications@github.com:

Currently the spec does not allow orchestration engines to request allocation of a specific IP address from the plugins. Are there any thoughts/proposals to support static IP addresses in CNI?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[dcbw]

If the requested IP address is not available for allocation (because it's outside of the given Subnet or has already been allocated to another pod) then the ADD request will fail and the orchestration runtime will receive an error that it can handle appropriately.

@asridharan did these replies answer your question? Anything else to clarify?

[asridharan]

My problem is that the host-local plugin is an anecdotal evidence of how static IP addressing be achieved with CNI. There is nothing in the spec that suggests that if an IPAM supports static IP addressing the way to pass the static IP request is through the CNI_ARGS, with key being "ip" .

The reason I am emphasizing this not being part of the standard is that container runtimes are agnostic of the underlying plugin and IPAM that is being used. This makes it hard for the container runtimes to handle static IP addressing requests to the IPAM as there is nothing standardized in the specification to achieve this.

[asridharan]

If ip=xxx CNI-arg is the way to go, then we should add it to the CNI spec.

[bboreham]

This makes it hard

I think I understand your preference for a spec change, but I'm struggling to understand what exactly is "hard".

Today, some plugins support that arg and some don't. If we add it to the spec, this situation will be exactly the same.

Perhaps you meant it is hard to discover the method. That would be an argument to add it to the documentation - some list of args that exist but are not in the spec.

Perhaps you meant it is hard to know whether any individual plugin supports this request. That is a more general point about inquiring the status and capabilities of a plugin. Doing this by bumping the spec version is a very coarse approach and will eventually run out of power.

[asridharan]

Thanks for the clarification @bboreham . Yes, when I say its hard for the container run-time I meant to say its hard to discover the method through which we can make the request to the plugin for a static IP address. Standardizing the invocation would make it more deterministic for the container run time.

I am not concerned with discovering the capabilities of the plugin. That is the responsibility of the operator at install/configuration time, for which the operator can rely on the documentation of the specific plugin.

[leodotcloud]

@asridharan

As mentioned by others, you are free to customize the provided sample IPAM plugin implementation for your needs instead of a SPEC change.

Just to give you another example, here is our custom IPAM plugin: https://github.com/rancher/rancher-cni-ipam. Summary: The IPAM plugin talks to our internal metadata server to fetch the IP address to be assigned to the container and returns it back to the invoking CNI network plugin. In your case you can use a similar approach to create a custom IPAM plugin to return a pre-determined IP address for your container.

If this answers your question, could you please close the issue? Otherwise further discussions/comments/questions are really welcome.

[asridharan]

@leodotcloud checkout the comment from @bboreham and my response to it. The issue is not the implementation of a static IP in a specific IPAM, the issue is for defining/standardizing the API to ask for a static IP. It could be through CNI args (as is done for host-local IPAM) or through the CNI config (add an explicit field for static IP, or define a behavior). Either one is fine. Bottomline this needs to be made part of the spec so that container runtimes have a standard way of requesting static IP addresses.

The issue is specifically from the standpoint of a container run time (I work on Mesos) and how the container run-time requests a static IP from "any" CNI plugin.

[asridharan]

CNI maintainers, can we get a resolution on this? I am fine with specifying static addresses by setting ip=<IP address in CNI_ARGS, but would be great if we can make it part of the spec.

[rosenhouse]

We discussed this a bit today at the maintainers meeting. This feels to me like a "capability", similar to the portMappings one already defined in our conventions doc.

[asridharan]

@rosenhouse that sounds reasonable to me.

[f1-outsourcing]

Where can I find how this should be configured, because I have problems getting this to work at the cni network plugin configuration as well as the application.

{
  "id": "influxdb",
  "cmd": null,
  "cpus": 0.5,
  "mem": 128,
  "disk": 0,
  "instances": 1,
  "acceptedResourceRoles": ["*"],
  "container": {
    "type": "MESOS",
    "volumes": [],
    "docker": {
      "image": "influxdb",
      "credential": null,
      "forcePullImage": false
    }
  },
  "ipAddress": {
    "networkName": "cni-bridge-test"
  },
"args": {"ips": ["192.168.0.100", "192.168.0.101"]},
    "healthChecks": [
    {
      "gracePeriodSeconds": 240,
      "intervalSeconds": 10,
      "timeoutSeconds": 10,
      "maxConsecutiveFailures": 10,
      "port": 8086,
      "path": "/ping",
      "protocol": "HTTP",
      "ignoreHttp1xx": false
    }
  ]
}

{
"name": "cni-bridge-test",
"type": "bridge",
"bridge": "mesos-cni0",
"isGateway": true,
"args":{
    "ips": ["192.168.0.100", "192.168.0.101"]
},
"ipam": {
    "type": "host-local",
    "subnet": "192.168.0.0/16",
    "routes": [
    { "dst":
      "0.0.0.0/0" }
    ]
  }
}

mesosphere-zookeeper-3.4.6-0.1.20141204175332.centos7.x86_64 mesos-1.4.1-2.0.1.x86_64 containernetworking-cni-0.5.1-1.el7.x86_64 marathon-1.4.9-1.0.668.el7.x86_6

[squeed]

Almost there, but the args block needs an additional cni block:

{
  "name": "cni-bridge-test",
  "args": {
    "cni": {
      "ips": ["192.168.0.100"]
    }
  }
}

[asridharan]

@squeed @f1-outsourcing the ability to configure static IP address is a limitation of Mesos and not CNI. Mesos only supports v0.2.0 of the spec and I think cni in the args dict as an acceptable value got added in v0.3.0 ? So not sure it would actually work with plugins, since they should be defaulting to v0.2.0.

[squeed]

Fortunately, the CNI plugins are not that picky - I think it will still work.

[rc979]

What is the latest recommendation on this topic?

I see that there is an ipam/static plugin which seems to work fine but is not part of the plugin releases, any particular reason for this?

[squeed]

There are two recommendations. The first is to use host-local, and pass the requested IP address either via the CNI_ARGS environment variable or via the args parameter in the configuration file.

The second is to use the static plugin, which has not yet been released but will be in the coming weeks.

[f1-outsourcing]

There are two recommendations. The first is to use host-local, and pass the requested IP address either via the CNI_ARGS environment variable or via the args parameter in the configuration file.

The second is to use the static plugin, which has not yet been released but will be in the coming weeks.

Do you know how I should put this in a marathon json configuration. It is very difficult any response from marathon support. The configuration below does not work (also without the single quotes). And putting args in the configuration file works

{
  "id": "/server",
  "user": "nobody",
  "cmd": "python -m SimpleHTTPServer 8080",
  "cpus": 0.1,
  "mem": 32,
  "disk": 0,
  "instances": 1,
  "acceptedResourceRoles": ["*"],
  "constraints": [["hostname","CLUSTER","m03.local"]],
  "backoffSeconds": 10,
  "networks": [ { "mode": "container", "name": "test-macvtap-cniip" } ],
  "env": { "CNI_ARGS" : "'IP=192.168.122.178'" }
}