Multiple Interfaces and IPs not detected in AWS VPC

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1153626

Launchpad details

affected_projects = ['cloud-init (Ubuntu)']
assignee = None
assignee_name = None
date_closed = 2023-05-05T09:02:40.095115+00:00
date_created = 2013-03-11T14:49:16.119357+00:00
date_fix_committed = 2023-05-05T09:02:40.095115+00:00
date_fix_released = 2023-05-05T09:02:40.095115+00:00
id = 1153626
importance = medium
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1153626
milestone = None
owner = smoser
owner_name = Scott Moser
private = False
status = fix_released
submitter = mikehale
submitter_name = Michael Hale
tags = ['aws', 'cloud-images', 'ec2-images', 'patch', 'vpc']
duplicates = []

Launchpad user Michael Hale(mikehale) wrote on 2013-03-11T14:49:16.119357+00:00

I booted a c1.medium with ami-3fec7956, 2 interfaces and 2 IPs per interface. I expected the 4 IPs to be configured on the instance when I logged in, however only the primary interface had a single IP assigned.

DHCP does not seem to support multiple IPs per interface so the alias IPs will probably need to be statically assigned based on the EC2 meta-data API (http://169.254.169.254/latest/meta-data/).

Additionally since network interfaces, and their IPs, can be moved between running instances in VPC, any solution should detect hot-plug events and not just setup things at boot time.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2013-03-11T20:00:42.938991+00:00

Thanks for taking the time to open a bug and make ubuntu better.

When multiple interfaces and IPs first came out, I looked at this, but yours is the first request for this.

I agree, I'd like to have a udev hook that would fire on interface adaded and then determine if it was in EC2, and if so, hit the EC2 metadata service and configure the device. That'd be smooth.

I'm more than happy to review and help out, but its not something that will be high priority on my plate at the moment.

[ubuntu-server-builder]

Launchpad user Michael Hale(mikehale) wrote on 2013-03-13T15:38:51.502544+00:00

Scott, thanks for the reply. This is not something I need at the moment, but it does seem like a good way for me to perhaps learn more about Ubuntu network configuration and udev rules. I did a bit of digging and found the following files which seem relevant:

/lib/udev/write_net_rules /lib/udev/rules.d/75-net-description.rules
/lib/udev/rules.d/75-persistent-net-generator.rules /etc/udev/rules.d/70-persistent-net.rules

Are there any other starting points I should be aware of?

[ubuntu-server-builder]

Launchpad user Michael Hale(mikehale) wrote on 2013-03-19T04:34:38.830383+00:00

My current thinking is that a udev rule will need to have a RUN hook that updates /etc/network/interfaces or at a minimum runs the appropriate ip commands based on what is discovered in the EC2 metadata service. It seems like updating /etc/network/interfaces would be good since ifup/ifdown rely on its configuration. According to the VPC docs (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#detach_eni) it appears that eth0 can not be removed from a running instance while eth1 may be removed or added while the machine is running. However, it is unclear if additional VIPs can be added to eth0 while the instance is running. I'm not sure how changes to the available IPs would be detected without polling the meta-data service.

[ubuntu-server-builder]

Launchpad user Michael Hale(mikehale) wrote on 2013-03-19T20:14:33.886895+00:00

Adding notes from IRC:

• use 'manual eth1' to avoid cloud-init blocking on boot waiting for interfaces
• /etc/network/interfaces originates from a hardcoded file in the image
• cloud-init places /etc/cloud/on-ec2 (or similar) flag file
• on interface add/remove have a rule that fires, checks ec2 flag file, then updates /etc/network/interfaces, and ifup/down

[ubuntu-server-builder]

Launchpad user Forest(foresto) wrote on 2013-11-26T02:44:14.789915+00:00

I'm in a similar situation: An ec2 instance with just one network interface, to which I have assigned a secondary IP address.

This bit from the AWS docs is interesting:

"If you are using Amazon Linux, the ec2-net-utils package can take care of this step for you. It configures additional network interfaces that you attach while the instance is running, refreshes secondary IP addresses during DHCP lease renewal, and updates the related routing rules."

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html#StepTwoConfigOS

Perhaps it would be worthwhile for cloud-init to do the same thing that Amazon's ec2-net-utils package does.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2013-11-26T14:36:26.643617+00:00

Forest, Thanks for the link. I hadn't seen that utility. It does appear to be doing basically what I was suggesting.

[ubuntu-server-builder]

Launchpad user Forest(foresto) wrote on 2013-11-26T17:47:52.086832+00:00

Sure. The part that stood out to me was "refreshes secondary IP addresses during DHCP lease renewal." I haven't examined Amazon's code, but that description reads like it solves the problem of automatically configuring additional IP addresses without requiring additional network interfaces.

[ubuntu-server-builder]

Launchpad user Alle(alessandro-demaria) wrote on 2013-12-07T10:24:55.891905+00:00

Hi Forest,

looking at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#ec2-net-utils

"DHCP script Whenever the network interface receives a new DHCP lease, this script queries the instance metadata for Elastic IP addresses. For each Elastic IP address, it adds a rule to the routing policy database to ensure that outbound traffic from that address uses the correct network interface. It also adds each private IP address to the network interface as a secondary address."

I think all they do is run a hook script on DHCP lease that configures the interface. And yes they add multiple ip addresses to the same interface.

I am looking at some sort of solution because I want to use some instances as LXC hosts and would love to give each container its own ip. So I think I will have to go down the route of using subinterfaces.

Will keep tracking this bug

Thank you Alessandro

[ubuntu-server-builder]

Launchpad user Alle(alessandro-demaria) wrote on 2013-12-07T22:37:47.150658+00:00

Hi,

I have actually adapted the files from the Amazon AMI which are working just fine. There is also an ansible file which tells where the files should be dropped. You might need to adjust a few things but it should all be working

There is only a problem with this bug (https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1235169) which has actually been solved and I am waiting for the updated package to be available. Otherwise the solution for this is to add the interfaces-eth1 configuration into the main /etc/network/interfaces file.

I am not good at packaging stuff so if you think this is useful it would be great if someone could wrap it up in a nice package. Myself I am ok with using ansible for now :)

Hope this helps Regards Alessandro

Launchpad attachments: Modified files from Amazon AMI ec2net package

[ubuntu-server-builder]

Launchpad user Alle(alessandro-demaria) wrote on 2013-12-07T22:38:45.689840+00:00

Re-uploading attachment as solution Launchpad attachments: Re-uploading attachment as solution

[ubuntu-server-builder]

Launchpad user Alle(alessandro-demaria) wrote on 2013-12-08T17:04:25.408954+00:00

Fixed some bugs. New version here https://github.com/ademaria/ubuntu-ec2net

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2013-12-10T17:19:31.640541+00:00

I'm attaching ec2-utils-0.4-1.19.amzn1.src.rpm as I retrieved it with: get_reference_source -p ec2-net-utils from inside ami-83e4bcea (amazon/amzn-ami-pv-2013.09.1.x86_64-ebs) today.

The license per spec file and headers is 'Apache License 2.0'.

Launchpad attachments: ec2-utils-0.4.1.19.amzn1.src.rpm

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2013-12-10T17:34:08.369975+00:00

adding also as a tarball of the rpmbuild dir after rpm -Uvh (jsut for convenience).

Launchpad attachments: tarball of rpmbuild after install of ec2-utils-0.4-1.19.amzn1.src.rpm

[ubuntu-server-builder]

Launchpad user Jason Dusek(solidsnack) wrote on 2015-03-31T17:49:24.981731+00:00

We're using a variant Alessandro's work to configure out instances.

https://github.com/instacart/ubuntu-ec2net

Reboot doesn't seem to refresh the interfaces list -- maybe there is another event hook that we need? I'm not terribly familiar with udev but would love to see this ironed out and ready for Ubuntu.

Installing the RPMs (via Alien) seemed to have no effect at all.

[ubuntu-server-builder]

Launchpad user Jason Antman(jason-jasonantman) wrote on 2015-04-07T12:12:01.893918+00:00

I'm using cloud-init on CentOS, but would really like this functionality as well. For the time being, I'll be trying to reuse the upstream AWS pieces.

[ubuntu-server-builder]

Launchpad user Naren Palepu(naren-j) wrote on 2016-05-19T18:19:21.944268+00:00

Hi

This is still the same for new instances on AWS ec2 ubuntu 144 . Floating i/p secondary eth0 not coming up automatically. IS there a fix in plan for ubuntu 14,15 ?

Thanks,

NP

[ubuntu-server-builder]

Launchpad user Gaëtan Duchaussois(gaetanfl) wrote on 2019-07-09T09:12:02.686064+00:00

Hi, I made a small patch to support multiple ips on one interface. Should work on several interfaces too even if I din't test it. I tested it on ec2 with ubuntu 18.04 and netplan export Launchpad attachments: ec2_multiple_ips.patch

[ubuntu-server-builder]

Launchpad user Ubuntu Foundations Team Bug Bot(crichton) wrote on 2019-07-09T16:23:31.205114+00:00

The attachment "ec2_multiple_ips.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[ubuntu-server-builder]

Launchpad user Leo Crawford(leo-leocrawford) wrote on 2019-07-22T20:45:17.087835+00:00

My initial tests with Gaëtan's patch suggests it disables DHCP for but doesn't set a route or DNS manually.

ci-info: +--------+------+----------------+-----------------+--------+-------------------+ ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | ci-info: +--------+------+----------------+-----------------+--------+-------------------+ ci-info: | eth0 | True | 172.16.255.107 | 255.255.254.0 | global | 06:94:b2:8b:c0:3e | ci-info: | eth0 | True | 172.16.254.112 | 255.255.254.0 | global | 06:94:b2:8b:c0:3e | ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . | ci-info: | lo | True | 172.27.221.33 | 255.255.255.255 | global | . | ci-info: +--------+------+----------------+-----------------+--------+-------------------+ ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++ ci-info: +-------+--------------+--------------+---------------+-----------+-------+ ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | ci-info: +-------+--------------+--------------+---------------+-----------+-------+ ci-info: | 0 | 0.0.0.0 | 172.16.254.1 | 0.0.0.0 | eth0 | UG | ci-info: | 1 | 172.16.254.0 | 0.0.0.0 | 255.255.254.0 | eth0 | U | ci-info: +-------+--------------+--------------+---------------+-----------+-------+

and

network: version: 2 ethernets: eth0: addresses:

• 172.16.255.107/23
• 172.16.254.112/23 dhcp6: true match: macaddress: 06:94:b2:8b:c0:3e set-name: eth0

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2019-07-23T20:21:59.570790+00:00

Hi folks, as a heads-up, there's some in-progress work on secondary IP addresses on primary NICs here: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/369792

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2023-05-05T09:02:34.577592+00:00

This was implemented in https://github.com/canonical/cloud-init/pull/114/files