Open dwilkie opened 1 year ago
Hi,
You also need a slightly different runonce.sh:
#!/bin/bash -x
# attach the ENI
aws ec2 attach-network-interface \
--region "$(/usr/bin/ec2-metadata -z | sed 's/placement: \(.*\).$/\1/')" \
--instance-id "$(/usr/bin/ec2-metadata -i | cut -d' ' -f2)" \
--device-index 1 \
--network-interface-id "${eni_id}"
# start SNAT
systemctl enable snat
systemctl start snat
And another aws_ami data call:
data "aws_ami" "this" {
most_recent = true
owners = ["amazon"]
filter {
name = "architecture"
values = ["x86_64"]
}
filter {
name = "root-device-type"
values = ["ebs"]
}
filter {
name = "name"
values = ["al2023-ami-*"]
}
filter {
name = "virtualization-type"
values = ["hvm"]
}
}
This was a pretty painful yak shaving experience and I am not sure if it is correct. But here are my working changes for Amazon Linux 2023.
Change the ENI role to the following:
resource "aws_iam_role_policy" "eni" {
role = aws_iam_role.this.name
name_prefix = "${var.prefix}-${var.environment}-nat-eni"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AttachNetworkInterface",
"ec2:ModifyInstanceAttribute"
],
"Resource": "*"
}
]
}
EOF
}
Change the data block for the AMI to:
data "aws_ami" "this" {
most_recent = true
owners = ["amazon"]
filter {
name = "architecture"
values = ["x86_64"]
}
filter {
name = "root-device-type"
values = ["ebs"]
}
filter {
name = "name"
values = ["al2023-ami-2023*"]
}
filter {
name = "virtualization-type"
values = ["hvm"]
}
}
In my outdated comment it picked the minimal AMI causing me to lose access with Systems Manager.
Change the runonce.sh to this:
#!/bin/bash -x
# Disable Source/Destination Check for the instance default interface
aws ec2 modify-instance-attribute --instance-id "$(/usr/bin/ec2-metadata -i | cut -d' ' -f2)" --no-source-dest-check
# attach the ENI
aws ec2 attach-network-interface \
--region "$(/usr/bin/ec2-metadata -z | sed 's/placement: \(.*\).$/\1/')" \
--instance-id "$(/usr/bin/ec2-metadata -i | cut -d' ' -f2)" \
--device-index 1 \
--network-interface-id "${eni_id}"
# Install IP tables its not available by default on Amazon Linux 2023 anymore
sudo yum install -y iptables-services
sudo systemctl enable iptables
sudo systemctl start iptables
# start SNAT
systemctl enable snat
systemctl start snat
Change snat.sh to, I opted not to swap around the default interfaces but instead just redirect traffic with iptables:
#!/bin/bash -x
# wait for ens6
while ! ip link show dev ens6; do
sleep 1
done
# make this a nat instance
echo "net.ipv4.ip_forward = 1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
# Configure iptables to forward packets
sudo iptables -t nat -A POSTROUTING -o ens5 -j MASQUERADE
sudo iptables -A FORWARD -i ens6 -o ens5 -j ACCEPT
sudo iptables -A FORWARD -i ens5 -o ens6 -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited
sudo service iptables save
# ensure these settings persist across reboots
echo "@reboot root iptables-restore < /etc/sysconfig/iptables" | sudo tee -a /etc/crontab
# wait for network connection
curl --retry 10 http://www.google.com
# re-establish connections
systemctl restart amazon-ssm-agent
PS: I am not sure why it worked with Amazon Linux 2, but I think both of the ENI's need to have source/destination check set to false. This library only sets the source destination check to false on the new ENI (the one without the public IP). AFAIK there is no way to set it up for public IPs so it has to be done from the instances. Please correct me if I am wrong!
In addition, you might also want to change runonce.sh attaching of the ENI to this:
# try to reattach the ENI (if you reboot, sometimes the old instance does not release the ENI in time)
max_attempts=10
attempt=0
while true; do
aws ec2 attach-network-interface \
--region "$(/usr/bin/ec2-metadata -z | sed 's/placement: \(.*\).$/\1/')" \
--instance-id "$(/usr/bin/ec2-metadata -i | cut -d' ' -f2)" \
--device-index 1 \
--network-interface-id "${eni_id}" && break
attempt=$((attempt + 1))
if [ "$attempt" -ge "$max_attempts" ]; then
echo "Maximum attempts reached. Initiating reboot."
sudo reboot
break
fi
echo "Attempt $attempt failed. Retrying..."
sleep 5 # waits for 5 seconds before retrying
done
Because sometimes, if a instance shutdown, and a new instance starts quickly, the ENI might have not been released from the previous instance. This causes the NAT instance to never work because it unable to attach the ENI.
@alexjeen Thanks! That fixed my issues.
The current snat.sh script doesn't work on Amazon Linux 2023.
Here's my first attempt at an alternative script for Amazon Linux 2023
There's a couple of areas which could use improvement such as:
ens5
andens6
If there is anyone else interested in having this module work with Amazon Linux 2023 comment here and i'll submit a PR.