Add support for Enhanced Networking to aws_instance resource.

[hashibot]

This issue was originally opened by @kwilczynski as hashicorp/terraform#7815. It was migrated here as part of the provider split. The original body of the issue is below.

---

Amazon at the moment supports two flavours of Enhanced Networking feature:

Overview and documentation.

• SR-IOV (Intel 82599 VF) http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sriov-networking.html
• Elastic Network Adapter (ENA) http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html

For any VPC-enabled and HVM-compatible EC2 instances.

We should allow users to enable Enhanced Network support and choose which flavour they would like to use. A very common use case, would be to run and span a Docker overlay network layer across multiple instances participating as hosts in the Docker Swarm. Such nodes do benefit from more linear and consistent performance which Enhanced Networking be bring, especially when latency is concerned and momentary spikes in such and "choking up" network throughput (e.g. exhausting the packer per-second rate, etc.) is not desired.

For this to work an AMI (Amazon Machine Image) needs to have moderately up-to-date Linux kernel and include supported version of both the ixgbevf[1] and ena[2] kernel driver, thus it would be prudent to to include latest drives and to enable support for the Enhanced Networking when creating (with Packer, etc.) custom AMIs for both the EBS and Instance Store type.

The Elastic Network Adapter is quite new (see the announcement) and it is not yet supported on Windows due to lack of available drivers for this platform (an on-going development effort by Amazon), plus the only instance type that supports ENA at the moment is X1, but hopefully Amazon would add ENA support to others later.

Worth noting is, that the Amazon Linux HVM-enabled AMIs already ship with both of the drivers included and have Enhanced Networking support enabled, thus are ready to use from the get-go.

API documentation:

• https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceAttribute.html
• https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_ModifyInstanceAttribute.html

Conceptually, this very similar to the way how the "SourceDestCheck" check is now supported in Terraform, as per: resource_aws_instance.go#L595-L603

Related to https://github.com/mitchellh/packer/issues/3688 in Packer.

1. Intel 82599 VF (SR-IOV) drivers.
2. Amazon Elastic Network Adapter (ENA) drivers.

[h4wkmoon]

It seems ENA is mandatory for m5 instances.

[tavisma]

Looks to be required for C5 as well

[tata2000]

Any expectations on that one?

[swibowo]

👍 on this feature, please.

Per AWS documentation (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html), here is the list of instances that use ENA driver for enhanced networking (definitely much more than just X1 instance as mentioned in this issue's original description).

C5, F1, G3, H1, I3, M5, P2, P3, R4, X1, and m4.16xlarge instances use the Elastic Network Adapter for enhanced networking.

I think it is desirable to have the ena-support flag configurable on AMI (aws_ami) and EC2 instance (aws_instance) resources.

[jmvbxx]

Has there been an update on this? Or, is the idea to just wait it out until ENA is automatically enabled on all instances?

[tavisma]

So the process to enable ENA is a bit odd, it doesn't seem like there is an API call to enable support on an AMI. The current process (verified with Amazon) has you booting up an instance, making sure the necessary drivers are installed, modifying the instance to support ENA with an API call and creating an AMI from that instance

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html

[jmvbxx]

Correct. The appropriate kernel has to be used and then and awscli enable command and then restart. My solution was a quick Ansible role to make the kernel adjustments and then manually enable. Luckily I didn't have many older instances that didn't come preenabled.

On Jan 30, 2018 12:42, "Tavis" notifications@github.com wrote:

So the process to enable ENA is a bit odd, it doesn't seem like there is an API call to enable support on an AMI. The current process (verified with Amazon) has you booting up an instance, making sure the necessary drivers are installed, modifying the instance to support ENA with an API call and creating an AMI from that instance

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ enhanced-networking-ena.html

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/terraform-providers/terraform-provider-aws/issues/206#issuecomment-361673882, or mute the thread https://github.com/notifications/unsubscribe-auth/AJryY-9SIrO-Rpo7_35FVuY33dXXIKZbks5tP1RvgaJpZM4N41Lk .

[raddy]

Would love this included!

[paolo-rendano]

Yes, I would also love that. What about launch configurations with autoscaling? I would expect to be able to set this also there. is it?

[keien]

There is also an EnaSupport field for ec2.Image (doc) aka terraform aws_ami which would be lovely to have supported as well - newer instance types like m5's won't let you launch from an AMI without this enabled, which is currently what happens when you use terraform to manage your AMIs.

[bflad]

The aws_ami resource will have a new ena_support argument in version 1.30.0 of the AWS provider, releasing later today. 👍

[samm-git]

@bflad we got a problem due to that PR. After provider upgrade plan decided to re-create all ami-s because ena_support in defaults was not matching ena_support from aws. I dont think its a right behavior and could be very dangerous.

[bflad]

Hi @samm-git 👋 Sorry you are running into unexpected behavior. I'd suggest opening a new bug report issue filling out all the details if you feel its a bug so its not sending a notification to everyone following this separate feature request issue. 👍

[samm-git]

Hi, thank you for reply. @bflad already did in HashCorp bugtracker, so probably you will be contacted soon )

[CpuID]

Just to clarify - is this only on aws_ami and not aws_instance right now...?

[ianomad]

Would be nice to get an update on this..

[tavisma]

We ended up using Packer to create customised images (which is useful for a host of other reasons)

[pdomagala]

Any update on this? M5 instances were introduced in 2017...

[github-actions[bot]]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[gregharvey]

As we're coming to upgrade instances this is going to become more of an issue for us. For now we'll have to do the ENA support step manually, which is less than ideal.

[notwaf]

Very much +1 as well here

[devi-labs]

+1 t3 instances can't be built using terraform because of this as well!

[manuelbernhardt]

+1 would be nice to have support for this

[easystevebro]

+1 on this one, looking at the documentation for v5.47.0, and comparing to the EC2 UI

I create a new launch template --> Specify Network Interface, ENA Express / ENA Express UDP is disabled.

I turn on the ENA Express using the UI

Perform a terraform plan, and it shows all the attributes are hidden except for attributes thats unrelated (description etc).

I have a look at aws_network_interface and it seems to be missing either.

The only way i can theorise doing this is to deploy the launch template with

lifecycle {
   ignore_changes = [ network_interfaces ]
}

Then manually modify the ENA Express setting, then 🙏 that i dont forget. Hopefully there is a better way to do this?