awslabs / amazon-eks-ami

Packer configuration for building a custom EKS AMI
https://awslabs.github.io/amazon-eks-ami/
MIT No Attribution
2.44k stars 1.15k forks source link

feat: Add argument to bootstrap.sh to configure node and kubelete for network proxy #1182

Closed jebbens closed 2 days ago

jebbens commented 1 year ago

What would you like to be added: I would like to add support for configuring a host and kubelet for use with a network proxy to the bootstrap.sh (or add another file to the Packer build that bootstrap.sh could trigger).

Why is this needed: Many of our customers operate clusters behind a network proxy. This requires additional configuration on the host, kubelet, etc. There is limited guidance on this process and it can require a LOT of time and troubleshooting to get it right. I"m wondering if we could handle this undifferentiated heavy lifting for customers by adding an argument, e.g. --proxy, that, when present, triggers some additional logic in the bootstrap.sh script to run. Here is some example code:

--BOUNDARY
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
exec > >(tee /var/log/pre-user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
set -x
PROXY_URL="${outbound_proxy_url}"
if [[ -n "$PROXY_URL" ]]; then
# Set NO_PROXY_LIST
EKS_IPv4_RANGE=${service_ipv4_cidr}
VPC_CIDR=${vpc_cidr}
if [[ "${cluster_endpoint_private_access}" == "true" ]]
then EKS_ENDPOINT="eks.${dns_suffix}"
fi
NO_PROXY_LIST=$EKS_IPv4_RANGE,$VPC_CIDR,localhost,127.0.0.1,169.254.169.254,.internal,${no_proxy_endpoints},$EKS_ENDPOINT
NO_PROXY_LIST=$(sed "s/,,/,/g" <<< "$NO_PROXY_LIST")
NO_PROXY_LIST=$(sed "s/,$//" <<< "$NO_PROXY_LIST")

# Set proxy for future processes
cloud-init-per instance env_proxy_config cat <<EOF >> /etc/profile.d/set_proxy.sh
export http_proxy="$PROXY_URL"
export https_proxy="$PROXY_URL"
export no_proxy="$NO_PROXY_LIST"
export HTTP_PROXY="$PROXY_URL"
export HTTPS_PROXY="$PROXY_URL"
export NO_PROXY="$NO_PROXY_LIST"
export AWS_DEFAULT_REGION="${region}"
EOF

# Create the Docker systemd directory
mkdir -p /etc/systemd/system/docker.service.d

# Configure Docker to use the proxy
cloud-init-per instance docker_proxy_config tee <<EOF /etc/systemd/system/docker.service.d/http-proxy.conf >/dev/null
[Service]
Environment="HTTP_PROXY=$PROXY_URL"
Environment="HTTPS_PROXY=$PROXY_URL"
Environment="NO_PROXY=$NO_PROXY_LIST"
EOF

# Configure the kubelet with the proxy
cloud-init-per instance kubelet_proxy_config tee <<EOF /etc/systemd/system/kubelet.service.d/http-proxy.conf >/dev/null
[Service]
Environment="HTTP_PROXY=$PROXY_URL"
Environment="HTTPS_PROXY=$PROXY_URL"
Environment="NO_PROXY=$NO_PROXY_LIST"
EOF

# Reload the daemon and restart Docker to reflect proxy configurations at launch of instance
cloud-init-per instance reload_daemon systemctl daemon-reload
cloud-init-per instance enable_docker systemctl enable --now --no-block docker

# https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-proxy-with-ssm-agent.html#ssm-agent-proxy-systemd
mkdir /etc/systemd/system/amazon-ssm-agent.service.d
cat <<EOF >> /etc/systemd/system/amazon-ssm-agent.service.d/override.conf
[Service]
Environment="http_proxy=$PROXY_URL"
Environment="https_proxy=$PROXY_URL"
Environment="no_proxy=169.254.169.254"
EOF

systemctl daemon-reload && systemctl restart amazon-ssm-agent

source /etc/profile.d/set_proxy.sh
fi

--BOUNDARY--
cartermckinnon commented 1 year ago

Overall sounds reasonable, feel free to open a PR if you've got a proposed implementation.

Our Docker support is on the way out, so we likely wouldn't merge the Docker portion.

cartermckinnon commented 2 days ago

Still open to a PR on this one, but going to close for now