search

aws / amazon-ecs-ami

Packer recipes for building the official ECS-optimized Amazon Linux AMIs
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html
Apache License 2.0
204 stars 47 forks source link

Docker upgraded at startup #306

Closed yongzhang closed 2 months ago

yongzhang commented 2 months ago

Summary

We expericed a big issue that EC2 instances cannot join ECS cluster because of docker verison upgrades when new instances launched.

Description

We run ECS optimized AMI with docker version 20.x and ecs-agent 1.82.x, e.g. amzn2-ami-ecs-hvm-2.0.20240328-x86_64-ebs.

We can see docker upgraded to 25.x at startup:

# docker version
Client:
 Version:           25.0.5
 API version:       1.44
 Go version:        go1.22.5
 Git commit:        5dc9bcc
 Built:             Thu Aug 22 17:25:26 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          25.0.6
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.22.5
  Git commit:       b08a51f
  Built:            Thu Aug 22 17:26:01 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.11
  GitCommit:        64b8a811b07ba6288238eefc14d898ee0b5b99ba
 runc:
  Version:          1.1.11
  GitCommit:        4bccb38cc9cf198d52bebf2b3a90cd14e7af8c06
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

We checked cloud-init logs and found this:

/var/log/cloud-init-output.log:

Cloud-init v. 19.3-46.amzn2.0.1 running 'modules:config' at Tue, 03 Sep 2024 00:34:30 +0000. Up 9.23 seconds.
Loaded plugins: priorities, update-motd, upgrade-helper
 --> 1:openssl-1.0.2k-24.amzn2.0.12.x86_64 from installed removed (updateinfo)
 --> 2:microcode_ctl-2.1-47.amzn2.4.17.x86_64 from installed removed (updateinfo)
 --> openssh-clients-7.4p1-22.amzn2.0.7.x86_64 from amzn2-core removed (updateinfo)
 --> libdb-utils-5.3.21-24.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo)
 --> 1:openssl-libs-1.0.2k-24.amzn2.0.12.x86_64 from installed removed (updateinfo)
 --> 1:grub2-2.06-14.amzn2.0.3.x86_64 from installed removed (updateinfo)
 --> cloud-init-19.3-46.amzn2.0.2.noarch from amzn2-core removed (updateinfo)
 --> 1:grub2-tools-2.06-14.amzn2.0.3.x86_64 from installed removed (updateinfo)
 --> 1:grub2-pc-modules-2.06-14.amzn2.0.3.noarch from installed removed (updateinfo)
 --> ecs-init-1.86.2-1.amzn2.x86_64 from amzn2extra-ecs removed (updateinfo)
 --> ca-certificates-2023.2.68-1.amzn2.0.1.noarch from amzn2-core removed (updateinfo)
 --> python-jwcrypto-0.4.2-1.amzn2.0.1.noarch from amzn2-core removed (updateinfo)
 --> openssh-7.4p1-22.amzn2.0.7.x86_64 from amzn2-core removed (updateinfo)
 --> amazon-ssm-agent-3.3.380.0-1.amzn2.x86_64 from amzn2-core removed (updateinfo)
 --> 1:grub2-tools-minimal-2.06-14.amzn2.0.3.x86_64 from installed removed (updateinfo)
 --> rpm-4.11.3-48.amzn2.0.3.x86_64 from installed removed (updateinfo)
 --> python-jinja2-2.7.2-3.amzn2.0.1.noarch from installed removed (updateinfo)
 --> libdb-5.3.21-24.amzn2.0.4.x86_64 from installed removed (updateinfo)
 --> 2:tar-1.26-35.amzn2.0.4.x86_64 from amzn2-core removed (updateinfo)
 --> openssh-7.4p1-22.amzn2.0.6.x86_64 from installed removed (updateinfo)
 --> 1:grub2-tools-minimal-2.06-14.amzn2.0.4.x86_64 from amzn2-core removed (updateinfo)
 --> libdb-5.3.21-24.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo)
 --> containerd-1.7.11-1.amzn2.0.1.x86_64 from @amzn2extra-docker removed (updateinfo)
 --> cyrus-sasl-lib-2.1.26-24.amzn2.x86_64 from installed removed (updateinfo)
 --> amazon-efs-utils-1.35.2-1.amzn2.noarch from @amzn2-core removed (updateinfo)
 --> amazon-ssm-agent-3.2.2303.0-1.amzn2.x86_64 from @amzn2-core removed (updateinfo)
 --> 1:grub2-tools-extra-2.06-14.amzn2.0.4.x86_64 from amzn2-core removed (updateinfo)
 --> python-jinja2-2.7.2-3.amzn2.0.2.noarch from amzn2-core removed (updateinfo)
 --> 1:grub2-pc-2.06-14.amzn2.0.3.x86_64 from installed removed (updateinfo)
 --> python2-rpm-4.11.3-48.amzn2.0.4.x86_64 from amzn2-core removed (updateinfo)
 --> python2-rpm-4.11.3-48.amzn2.0.3.x86_64 from installed removed (updateinfo)
 --> rpm-libs-4.11.3-48.amzn2.0.3.x86_64 from installed removed (updateinfo)
 --> 1:grub2-pc-modules-2.06-14.amzn2.0.4.noarch from amzn2-core removed (updateinfo)
 --> amazon-efs-utils-2.0.4-1.amzn2.x86_64 from amzn2-core removed (updateinfo)
 --> krb5-libs-1.15.1-55.amzn2.2.6.x86_64 from installed removed (updateinfo)
 --> ca-certificates-2023.2.64-1.amzn2.0.1.noarch from installed removed (updateinfo)
 --> 2:tar-1.26-35.amzn2.0.3.x86_64 from installed removed (updateinfo)
 --> 1:grub2-common-2.06-14.amzn2.0.4.noarch from amzn2-core removed (updateinfo)
 --> 1:grub2-efi-x64-ec2-2.06-14.amzn2.0.4.x86_64 from amzn2-core removed (updateinfo)
 --> 1:grub2-efi-x64-ec2-2.06-14.amzn2.0.3.x86_64 from installed removed (updateinfo)
 --> rpm-plugin-systemd-inhibit-4.11.3-48.amzn2.0.4.x86_64 from amzn2-core removed (updateinfo)
 --> curl-8.3.0-1.amzn2.0.7.x86_64 from amzn2-core removed (updateinfo)
 --> rpm-build-libs-4.11.3-48.amzn2.0.4.x86_64 from amzn2-core removed (updateinfo)
 --> rpm-build-libs-4.11.3-48.amzn2.0.3.x86_64 from installed removed (updateinfo)
 --> cyrus-sasl-lib-2.1.26-24.amzn2.0.1.x86_64 from amzn2-core removed (updateinfo)
 --> 1:grub2-2.06-14.amzn2.0.4.x86_64 from amzn2-core removed (updateinfo)
 --> rpm-libs-4.11.3-48.amzn2.0.4.x86_64 from amzn2-core removed (updateinfo)
 --> ecs-init-1.82.1-1.amzn2.x86_64 from installed removed (updateinfo)
 --> 1:grub2-common-2.06-14.amzn2.0.3.noarch from installed removed (updateinfo)
 --> libcurl-8.3.0-1.amzn2.0.7.x86_64 from amzn2-core removed (updateinfo)
 --> rpm-plugin-systemd-inhibit-4.11.3-48.amzn2.0.3.x86_64 from installed removed (updateinfo)
 --> curl-8.3.0-1.amzn2.0.6.x86_64 from installed removed (updateinfo)
 --> 2:microcode_ctl-2.1-47.amzn2.4.20.x86_64 from amzn2-core removed (updateinfo)
 --> 1:grub2-tools-2.06-14.amzn2.0.4.x86_64 from amzn2-core removed (updateinfo)
 --> rpm-4.11.3-48.amzn2.0.4.x86_64 from amzn2-core removed (updateinfo)
 --> openssh-server-7.4p1-22.amzn2.0.6.x86_64 from installed removed (updateinfo)
 --> 1:openssl-1.0.2k-24.amzn2.0.13.x86_64 from amzn2-core removed (updateinfo)
 --> openssh-clients-7.4p1-22.amzn2.0.6.x86_64 from installed removed (updateinfo)
 --> libcurl-8.3.0-1.amzn2.0.6.x86_64 from installed removed (updateinfo)
 --> 1:grub2-pc-2.06-14.amzn2.0.4.x86_64 from amzn2-core removed (updateinfo)
 --> libdb-utils-5.3.21-24.amzn2.0.4.x86_64 from installed removed (updateinfo)
 --> 1:openssl-libs-1.0.2k-24.amzn2.0.13.x86_64 from amzn2-core removed (updateinfo)
 --> cloud-init-19.3-46.amzn2.0.1.noarch from installed removed (updateinfo)
 --> krb5-libs-1.15.1-55.amzn2.2.8.x86_64 from amzn2-core removed (updateinfo)
 --> 1:grub2-tools-efi-2.06-14.amzn2.0.4.x86_64 from amzn2-core removed (updateinfo)
 --> openssh-server-7.4p1-22.amzn2.0.7.x86_64 from amzn2-core removed (updateinfo)
 --> python-jwcrypto-0.4.2-1.amzn2.noarch from installed removed (updateinfo)
 --> containerd-1.7.20-1.amzn2.0.1.x86_64 from amzn2extra-ecs removed (updateinfo)
14 package(s) needed (+0 related) for security, out of 49 available
Resolving Dependencies
--> Running transaction check
---> Package bind-export-libs.x86_64 32:9.11.4-26.P2.amzn2.13.5 will be updated
---> Package bind-export-libs.x86_64 32:9.11.4-26.P2.amzn2.13.8 will be an update
---> Package docker.x86_64 0:20.10.25-1.amzn2.0.4 will be updated
---> Package docker.x86_64 0:25.0.6-1.amzn2.0.2 will be an update
---> Package ecs-service-connect-agent.x86_64 0:v1.27.3.0-1.amzn2 will be updated
---> Package ecs-service-connect-agent.x86_64 0:v1.29.6.1-1.amzn2 will be an update
---> Package glib2.x86_64 0:2.56.1-9.amzn2.0.7 will be updated
---> Package glib2.x86_64 0:2.56.1-9.amzn2.0.8 will be an update
---> Package glibc.x86_64 0:2.26-63.amzn2.0.1 will be updated
---> Package glibc.x86_64 0:2.26-64.amzn2.0.2 will be an update
---> Package glibc-all-langpacks.x86_64 0:2.26-63.amzn2.0.1 will be updated
---> Package glibc-all-langpacks.x86_64 0:2.26-64.amzn2.0.2 will be an update
---> Package glibc-common.x86_64 0:2.26-63.amzn2.0.1 will be updated
---> Package glibc-common.x86_64 0:2.26-64.amzn2.0.2 will be an update
---> Package glibc-locale-source.x86_64 0:2.26-63.amzn2.0.1 will be updated
---> Package glibc-locale-source.x86_64 0:2.26-64.amzn2.0.2 will be an update
---> Package glibc-minimal-langpack.x86_64 0:2.26-63.amzn2.0.1 will be updated
---> Package glibc-minimal-langpack.x86_64 0:2.26-64.amzn2.0.2 will be an update
---> Package less.x86_64 0:458-9.amzn2.0.3 will be updated
---> Package less.x86_64 0:458-9.amzn2.0.4 will be an update
---> Package libcrypt.x86_64 0:2.26-63.amzn2.0.1 will be updated
---> Package libcrypt.x86_64 0:2.26-64.amzn2.0.2 will be an update
---> Package libnghttp2.x86_64 0:1.41.0-1.amzn2.0.4 will be updated
---> Package libnghttp2.x86_64 0:1.41.0-1.amzn2.0.5 will be an update
---> Package python3.x86_64 0:3.7.16-1.amzn2.0.4 will be updated
---> Package python3.x86_64 0:3.7.16-1.amzn2.0.6 will be an update
---> Package python3-libs.x86_64 0:3.7.16-1.amzn2.0.4 will be updated
---> Package python3-libs.x86_64 0:3.7.16-1.amzn2.0.6 will be an update
--> Processing Conflict: docker-25.0.6-1.amzn2.0.2.x86_64 conflicts ecs-init < 1.86.1
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package ecs-init.x86_64 0:1.82.1-1.amzn2 will be updated
---> Package ecs-init.x86_64 0:1.86.2-1.amzn2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                 Arch   Version                    Repository      Size
================================================================================
Updating:
 bind-export-libs        x86_64 32:9.11.4-26.P2.amzn2.13.8 amzn2-core     1.1 M
 docker                  x86_64 25.0.6-1.amzn2.0.2         amzn2extra-ecs  45 M
 ecs-init                x86_64 1.86.2-1.amzn2             amzn2extra-ecs  28 M
 ecs-service-connect-agent
                         x86_64 v1.29.6.1-1.amzn2          amzn2extra-ecs  44 M
 glib2                   x86_64 2.56.1-9.amzn2.0.8         amzn2-core     2.4 M
 glibc                   x86_64 2.26-64.amzn2.0.2          amzn2-core     3.3 M
 glibc-all-langpacks     x86_64 2.26-64.amzn2.0.2          amzn2-core     7.0 M
 glibc-common            x86_64 2.26-64.amzn2.0.2          amzn2-core     774 k
 glibc-locale-source     x86_64 2.26-64.amzn2.0.2          amzn2-core     3.2 M
 glibc-minimal-langpack  x86_64 2.26-64.amzn2.0.2          amzn2-core      33 k
 less                    x86_64 458-9.amzn2.0.4            amzn2-core     119 k
 libcrypt                x86_64 2.26-64.amzn2.0.2          amzn2-core      53 k
 libnghttp2              x86_64 1.41.0-1.amzn2.0.5         amzn2-core      73 k
 python3                 x86_64 3.7.16-1.amzn2.0.6         amzn2-core      73 k
 python3-libs            x86_64 3.7.16-1.amzn2.0.6         amzn2-core     9.5 M

Transaction Summary
================================================================================
Upgrade  15 Packages

Total download size: 145 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
--------------------------------------------------------------------------------
Total                                              111 MB/s | 145 MB  00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : glibc-minimal-langpack-2.26-64.amzn2.0.2.x86_64             1/30
  Updating   : glibc-common-2.26-64.amzn2.0.2.x86_64                       2/30
  Updating   : glibc-2.26-64.amzn2.0.2.x86_64                              3/30
  Updating   : libcrypt-2.26-64.amzn2.0.2.x86_64                           4/30
  Updating   : python3-libs-3.7.16-1.amzn2.0.6.x86_64                      5/30
  Updating   : python3-3.7.16-1.amzn2.0.6.x86_64                           6/30
  Updating   : docker-25.0.6-1.amzn2.0.2.x86_64                            7/30
  Updating   : ecs-init-1.86.2-1.amzn2.x86_64                              8/30
  Updating   : glibc-all-langpacks-2.26-64.amzn2.0.2.x86_64                9/30
  Updating   : libnghttp2-1.41.0-1.amzn2.0.5.x86_64                       10/30
  Updating   : glibc-locale-source-2.26-64.amzn2.0.2.x86_64               11/30
  Updating   : 32:bind-export-libs-9.11.4-26.P2.amzn2.13.8.x86_64         12/30
  Updating   : less-458-9.amzn2.0.4.x86_64                                13/30
  Updating   : glib2-2.56.1-9.amzn2.0.8.x86_64                            14/30
  Updating   : ecs-service-connect-agent-v1.29.6.1-1.amzn2.x86_64         15/30
  Cleanup    : glibc-locale-source-2.26-63.amzn2.0.1.x86_64               16/30
  Cleanup    : glibc-all-langpacks-2.26-63.amzn2.0.1.x86_64               17/30
  Cleanup    : ecs-service-connect-agent-v1.27.3.0-1.amzn2.x86_64         18/30
  Cleanup    : python3-3.7.16-1.amzn2.0.4.x86_64                          19/30
  Cleanup    : python3-libs-3.7.16-1.amzn2.0.4.x86_64                     20/30
  Cleanup    : ecs-init-1.82.1-1.amzn2.x86_64                             21/30
  Cleanup    : docker-20.10.25-1.amzn2.0.4.x86_64                         22/30
  Cleanup    : libcrypt-2.26-63.amzn2.0.1.x86_64                          23/30
  Cleanup    : glib2-2.56.1-9.amzn2.0.7.x86_64                            24/30
  Cleanup    : less-458-9.amzn2.0.3.x86_64                                25/30
  Cleanup    : 32:bind-export-libs-9.11.4-26.P2.amzn2.13.5.x86_64         26/30
  Cleanup    : libnghttp2-1.41.0-1.amzn2.0.4.x86_64                       27/30
  Cleanup    : glibc-minimal-langpack-2.26-63.amzn2.0.1.x86_64            28/30
  Cleanup    : glibc-2.26-63.amzn2.0.1.x86_64                             29/30
  Cleanup    : glibc-common-2.26-63.amzn2.0.1.x86_64                      30/30
  Verifying  : ecs-service-connect-agent-v1.29.6.1-1.amzn2.x86_64          1/30
  Verifying  : glibc-all-langpacks-2.26-64.amzn2.0.2.x86_64                2/30
  Verifying  : ecs-init-1.86.2-1.amzn2.x86_64                              3/30
  Verifying  : libnghttp2-1.41.0-1.amzn2.0.5.x86_64                        4/30
  Verifying  : glibc-locale-source-2.26-64.amzn2.0.2.x86_64                5/30
  Verifying  : 32:bind-export-libs-9.11.4-26.P2.amzn2.13.8.x86_64          6/30
  Verifying  : docker-25.0.6-1.amzn2.0.2.x86_64                            7/30
  Verifying  : glibc-2.26-64.amzn2.0.2.x86_64                              8/30
  Verifying  : less-458-9.amzn2.0.4.x86_64                                 9/30
  Verifying  : python3-libs-3.7.16-1.amzn2.0.6.x86_64                     10/30
  Verifying  : python3-3.7.16-1.amzn2.0.6.x86_64                          11/30
  Verifying  : libcrypt-2.26-64.amzn2.0.2.x86_64                          12/30
  Verifying  : glibc-minimal-langpack-2.26-64.amzn2.0.2.x86_64            13/30
  Verifying  : glibc-common-2.26-64.amzn2.0.2.x86_64                      14/30
  Verifying  : glib2-2.56.1-9.amzn2.0.8.x86_64                            15/30
  Verifying  : glibc-common-2.26-63.amzn2.0.1.x86_64                      16/30
  Verifying  : glibc-2.26-63.amzn2.0.1.x86_64                             17/30
  Verifying  : python3-libs-3.7.16-1.amzn2.0.4.x86_64                     18/30
  Verifying  : python3-3.7.16-1.amzn2.0.4.x86_64                          19/30
  Verifying  : glibc-all-langpacks-2.26-63.amzn2.0.1.x86_64               20/30
  Verifying  : libnghttp2-1.41.0-1.amzn2.0.4.x86_64                       21/30
  Verifying  : 32:bind-export-libs-9.11.4-26.P2.amzn2.13.5.x86_64         22/30
  Verifying  : glib2-2.56.1-9.amzn2.0.7.x86_64                            23/30
  Verifying  : glibc-minimal-langpack-2.26-63.amzn2.0.1.x86_64            24/30
  Verifying  : libcrypt-2.26-63.amzn2.0.1.x86_64                          25/30
  Verifying  : docker-20.10.25-1.amzn2.0.4.x86_64                         26/30
  Verifying  : ecs-init-1.82.1-1.amzn2.x86_64                             27/30
  Verifying  : less-458-9.amzn2.0.3.x86_64                                28/30
  Verifying  : ecs-service-connect-agent-v1.27.3.0-1.amzn2.x86_64         29/30
  Verifying  : glibc-locale-source-2.26-63.amzn2.0.1.x86_64               30/30

Updated:
  bind-export-libs.x86_64 32:9.11.4-26.P2.amzn2.13.8
  docker.x86_64 0:25.0.6-1.amzn2.0.2
  ecs-init.x86_64 0:1.86.2-1.amzn2
  ecs-service-connect-agent.x86_64 0:v1.29.6.1-1.amzn2
  glib2.x86_64 0:2.56.1-9.amzn2.0.8
  glibc.x86_64 0:2.26-64.amzn2.0.2
  glibc-all-langpacks.x86_64 0:2.26-64.amzn2.0.2
  glibc-common.x86_64 0:2.26-64.amzn2.0.2
  glibc-locale-source.x86_64 0:2.26-64.amzn2.0.2
  glibc-minimal-langpack.x86_64 0:2.26-64.amzn2.0.2
  less.x86_64 0:458-9.amzn2.0.4
  libcrypt.x86_64 0:2.26-64.amzn2.0.2
  libnghttp2.x86_64 0:1.41.0-1.amzn2.0.5
  python3.x86_64 0:3.7.16-1.amzn2.0.6
  python3-libs.x86_64 0:3.7.16-1.amzn2.0.6

Complete!

^ This confirmed docker was upgraded to 25.x.

/var/log/cloud-init.log:

Sep 03 00:34:31 cloud-init[3312]: util.py[DEBUG]: Running command ['yum', '-t', '-y', '--exclude=kernel', '--exclude=nvidia*', '--exclude=cuda*', '--security', '--sec-severity=critical', '--sec-severity=important', 'upgrade'] with allowed return codes [0] (shell=False, capture=False)

/etc/cloud/cloud.cfg:

repo_upgrade: security
repo_upgrade_exclude:
 - kernel
 - nvidia*
 - cuda*

cloud_config_modules:
...
 - package-update-upgrade-install
...

So it seems like either docker was upgraded by cloud-init directly or upgraded as dependencies by other packages.

In the other word, even docker version was installed by a specific version in the AMI, like this, but docker version can still be upgraded to break things.

Expected Behavior

Observed Behavior

Environment Details

Supporting Log Snippets

singholt commented 2 months ago

Hi @yongzhang , Thank you for reaching out! This is the expected behavior on AL2 instances, since docker 25 is a security update.

If you'd like to disable security updates at launch, please refer to this Amazon Linux FAQ. (search for Q: How do I disable the automatic installation of critical and important security updates on initial launch?).

To further help ECS customers from running into surprises, ECS AMIs beginning June 12, 2024 have the security updates-at-boot feature disabled by default: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html#ecs-optimized-AMI-security-changes

Please let us know if you have further questions. Thanks!

yongzhang commented 2 months ago

@singholt

I can still find this in newer versions of AMI:

repo_upgrade: security

image

image

EDITTED:

Ok, I found the overriden in cloud.cfg.d:

# cat 90_ecs.cfg
#cloud-config
repo_upgrade: none

system_info:
  default_user:
    groups: [ "wheel", "docker" ]

Cool! Thanks.