search

awslabs / amazon-eks-ami

Packer configuration for building a custom EKS AMI
https://awslabs.github.io/amazon-eks-ami/
MIT No Attribution
2.38k stars 1.12k forks source link

amazon-ebs: fatal error: An error occurred (404) when calling the HeadObject operation: Key "1.30/2022-06-06/bin/linux/amd64/kubelet" does not exist #1839

Closed eravindar12 closed 3 weeks ago

eravindar12 commented 3 weeks ago

What happened:

I am attempting to use the AL2023 base AMI for EKS optimized with the AMI Release v20240531. However, I am encountering a "fatal error: An error occurred (404) when calling the HeadObject operation: Key "1.30/2022-06-06/bin/linux/amd64/kubelet" does not exist" This appears to be a bug.

Do you have any suggestions for a workaround to fix this issue?

Error details:

2024-06-06T19:54:42Z:     amazon-ebs: Downloading binaries from: s3://amazon-eks
2024-06-06T19:54:43Z:     amazon-ebs: AWS cli present - using it to copy binaries from s3.
2024-06-06T19:54:44Z:     amazon-ebs: fatal error: An error occurred (404) when calling the HeadObject operation: Key "1.30/2022-06-06/bin/linux/amd64/kubelet" does not exist
2024-06-06T19:54:44Z: ==> amazon-ebs: Provisioning step had errors: Running the cleanup provisioner, if present...
2024-06-06T19:54:44Z: ==> amazon-ebs: Terminating the source AWS instance...
2024-06-06T19:55:44Z: ==> amazon-ebs: Cleaning up any extra volumes...
2024-06-06T19:55:45Z: ==> amazon-ebs: No volumes to clean up, skipping
2024-06-06T19:55:45Z: ==> amazon-ebs: Deleting temporary security group...
2024-06-06T19:55:45Z: ==> amazon-ebs: Deleting temporary keypair...
2024-06-06T19:55:45Z: Build 'amazon-ebs' errored after 2 minutes 2 seconds: Script exited with non-zero exit status: 1. Allowed exit codes are: [0]
==> Wait completed after 2 minutes 2 seconds
==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Script exited with non-zero exit status: 1. Allowed exit codes are: [0]
==> Builds finished but no artifacts were created.
make[1]: *** [Makefile:95: k8s] Error 1

templates/install-worker.sh

################################################################################
### Kubernetes #################################################################
################################################################################

sudo mkdir -p /etc/kubernetes/manifests
sudo mkdir -p /var/lib/kubernetes
sudo mkdir -p /var/lib/kubelet
sudo mkdir -p /opt/cni/bin

echo "Downloading binaries from: s3://$BINARY_BUCKET_NAME"
S3_DOMAIN="amazonaws.com"
if [ "$BINARY_BUCKET_REGION" = "cn-north-1" ] || [ "$BINARY_BUCKET_REGION" = "cn-northwest-1" ]; then
  S3_DOMAIN="amazonaws.com.cn"
elif [ "$BINARY_BUCKET_REGION" = "us-iso-east-1" ] || [ "$BINARY_BUCKET_REGION" = "us-iso-west-1" ]; then
  S3_DOMAIN="c2s.ic.gov"
elif [ "$BINARY_BUCKET_REGION" = "us-isob-east-1" ]; then
  S3_DOMAIN="sc2s.sgov.gov"
fi
S3_URL_BASE="https://$BINARY_BUCKET_NAME.s3.$BINARY_BUCKET_REGION.$S3_DOMAIN/$KUBERNETES_VERSION/$KUBERNETES_BUILD_DATE/bin/linux/$ARCH"
S3_PATH="s3://$BINARY_BUCKET_NAME/$KUBERNETES_VERSION/$KUBERNETES_BUILD_DATE/bin/linux/$ARCH"

BINARIES=(
  kubelet
)
for binary in ${BINARIES[*]}; do
  if [[ -n "$AWS_ACCESS_KEY_ID" ]]; then
    echo "AWS cli present - using it to copy binaries from s3."
    aws s3 cp --region $BINARY_BUCKET_REGION $S3_PATH/$binary .
    aws s3 cp --region $BINARY_BUCKET_REGION $S3_PATH/$binary.sha256 .
  else
    echo "AWS cli missing - using wget to fetch binaries from s3. Note: This won't work for private bucket."
    sudo wget $S3_URL_BASE/$binary
    sudo wget $S3_URL_BASE/$binary.sha256
  fi
  sudo sha256sum -c $binary.sha256
  sudo chmod +x $binary
  sudo mv $binary /usr/bin/
done

sudo rm ./*.sha256

template.json

    {
      "type": "shell",
      "remote_folder": "{{ user `remote_folder`}}",
      "script": "{{template_dir}}/provisioners/install-worker.sh",
      "environment_vars": [
        "AWS_ACCESS_KEY_ID={{user `aws_access_key_id`}}",
        "AWS_SECRET_ACCESS_KEY={{user `aws_secret_access_key`}}",
        "AWS_SESSION_TOKEN={{user `aws_session_token`}}",
        "BINARY_BUCKET_NAME={{user `binary_bucket_name`}}",
        "BINARY_BUCKET_REGION={{user `binary_bucket_region`}}",
        "CONTAINERD_VERSION={{user `containerd_version`}}",
        "KUBERNETES_BUILD_DATE={{user `kubernetes_build_date`}}",
        "KUBERNETES_VERSION={{user `kubernetes_version`}}",
        "RUNC_VERSION={{user `runc_version`}}",
        "SSM_AGENT_VERSION={{user `ssm_agent_version`}}",
        "WORKING_DIR={{user `working_dir`}}"
      ],"execute_command": "{{.Vars}} bash '{{.Path}}'"
    },

Environment: EKS

cartermckinnon commented 3 weeks ago

What make command are you running?

eravindar12 commented 3 weeks ago

@cartermckinnon JFYR, I am running the make command as shown below.

make k8s=1.30 os_distro=al2023 aws_region="us-east-1" source_ami_id="ami-xxxx" source_ami_owners=xxxx source_ami_filter_name="CIS Amazon Linux 2023 Benchmark - Level 1 - v05 -prod-xxxxx" kubernetes_build_date="2022-06-06" kubernetes_version="1.30" subnet_id="subnet-xxxx" vpc_id="vpc-xxxx"

cartermckinnon commented 3 weeks ago

This is the problem:

kubernetes_build_date="2022-06-06"

There are no 1.30 binaries from 2022 😄 you can find the latest ones for 1.30 using this script: https://github.com/awslabs/amazon-eks-ami/blob/main/hack/latest-binaries.sh

eravindar12 commented 3 weeks ago

@cartermckinnon Thank you for correcting the build date; now it's able to download. However, I'm encountering a "permission denied" error.

Do you have any suggestions on why I'm getting this permission denied?

amazon-ebs: /tmp/script_5266.sh: line 225: /tmp/worker/shared/bin/imds: Permission denied

2024-06-06T23:13:27Z:     amazon-ebs: Downloading binaries from: s3://amazon-eks
2024-06-06T23:13:27Z:     amazon-ebs: AWS cli present - using it to copy binaries from s3.
2024-06-06T23:13:30Z:     amazon-ebs: download: s3://amazon-eks/1.30.0/2024-05-12/bin/linux/amd64/kubelet to ./kubelet
2024-06-06T23:13:31Z:     amazon-ebs: download: s3://amazon-eks/1.30.0/2024-05-12/bin/linux/amd64/kubelet.sha256 to ./kubelet.sha256
2024-06-06T23:13:31Z:     amazon-ebs: kubelet: OK
2024-06-06T23:13:32Z:     amazon-ebs: AWS cli present - using it to copy ecr-credential-provider from s3.
2024-06-06T23:13:33Z:     amazon-ebs: download: s3://amazon-eks/1.30.0/2024-05-12/bin/linux/amd64/ecr-credential-provider to ./ecr-credential-provider
2024-06-06T23:13:34Z:     amazon-ebs: amazon-ssm-agent.x86_64                3.3.380.0-1.amzn2023               @System
2024-06-06T23:13:34Z:     amazon-ebs: amazon-ssm-agent already present - skipping install
2024-06-06T23:13:34Z:     amazon-ebs: /tmp/script_5266.sh: line 225: /tmp/worker/shared/bin/imds: Permission denied
2024-06-06T23:13:34Z: ==> amazon-ebs: Provisioning step had errors: Running the cleanup provisioner, if present...
2024-06-06T23:13:34Z: ==> amazon-ebs: Terminating the source AWS instance...
2024-06-06T23:15:05Z: ==> amazon-ebs: Cleaning up any extra volumes...
2024-06-06T23:15:05Z: ==> amazon-ebs: No volumes to clean up, skipping
2024-06-06T23:15:05Z: ==> amazon-ebs: Deleting temporary security group...
2024-06-06T23:15:05Z: ==> amazon-ebs: Deleting temporary keypair...
2024-06-06T23:15:06Z: Build 'amazon-ebs' errored after 2 minutes 43 seconds: Script exited with non-zero exit status: 126. Allowed exit codes are: [0]
cartermckinnon commented 3 weeks ago

It's because you're using a CIS base image, same problem as here: https://github.com/awslabs/amazon-eks-ami/issues/1838#issuecomment-2153515642