Generic Linux kernel incompatibile with v2.3.1569.0?

nouse commented 4 years ago

We have some ec2 instances running on 4.4 generic kernel and some on running on 4.4 aws verion, system OS are both Ubuntu 16.04.

Last Friday after updating ssm agent to v2.3.1569.0, we found instances running generic kernel stop responding after system reboot, system log obtained from EC2 console only contains information before system reboot, but instances running aws kernel weren't affected. Rebooting instances didn't work, we terminated and recreated instances to solve the problem, and we restrict agent version to stay at 2.3.978.0.

2020/08/14 13:24:21Z: Amazon SSM Agent v2.3.1569.0 is running
2020/08/14 13:24:21Z: OsProductName: Ubuntu
2020/08/14 13:24:21Z: OsVersion: 16.04
         Stopping Authenticate and Authorize Users to Run Privileged Tasks...
         Stopping ACPI event daemon...
[[0;32m  OK  [0m] Stopped target Graphical Interface.
[[0;32m  OK  [0m] Stopped target Timers.
[[0;32m  OK  [0m] Stopped Daily apt upgrade and clean activities.
[[0;32m  OK  [0m] Stopped Daily Cleanup of Temporary Directories.
[[0;32m  OK  [0m] Stopped Message of the Day.
         Stopping Accounts Service...
         Stopping OpenTelemetry Collector Contrib...
[[0;32m  OK  [0m] Stopped target Cloud-init target.
[[0;32m  OK  [0m] Stopped Execute cloud user/final scripts.
[[0;32m  OK  [0m] Stopped Apply the settings specified in cloud-config.
[[0;32m  OK  [0m] Stopped target Cloud-config availability.
[[0;32m  OK  [0m] Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.
[[0;32m  OK  [0m] Stopped target Multi-User System.
         Stopping Regular background program processing daemon...
         Stopping node_exporter.service...
         Stopping LSB: Start/stop sysstat's sadc...
         Stopping LSB: daemon to balance interrupts for SMP systems...
         Stopping LSB: Set the CPU Frequency Scaling governor to "ondemand"...
         Stopping Unattended Upgrades Shutdown...
         Stopping Service for snap applicati...on-ssm-agent.amazon-ssm-agent...
         Stopping rabbitmq-exporter.service...
         Stopping rn-misc-exporter.service...
         Stopping sensorcore_registration.service...
         Stopping LSB: MD monitoring daemon...
[[0;32m  OK  [0m] Stopped Wait until snapd is fully seeded.
         Stopping filebeat.service...
         Stopping D-Bus System Message Bus...
         Stopping ffmpeg_server.service...
         Stopping LSB: Set up cgroupfs mounts....
         Stopping iptables_exporter.service...
         Stopping sensorcore_endpoint.service...
         Stopping LXD - container startup/shutdown...
         Stopping Deferred execution scheduler...
[[0;32m  OK  [0m] Stopped ssh-host-keys.service.
         Stopping OpenBSD Secure Shell server...
         Stopping LSB: automatic crash report generation...
         Stopping spooler.service...
[[0;32m  OK  [0m] Stopped target Login Prompts.
         Stopping Getty on tty1...
         Stopping rabbitmq.service...
         Stopping journalbeat.service...
         Stopping FUSE filesystem for LXC...
         Stopping Serial Getty on ttyS0...
         Stopping LSB: Record successful boot for GRUB...
         Stopping Snap Daemon...
         Stopping AWS X-Ray Daemon...
[[0;32m  OK  [0m] Stopped Daily apt download activities.
[[0;32m  OK  [0m] Unmounted /var/lib/lxcfs.
[[0;32m  OK  [0m] Stopped ACPI event daemon.
[[0;32m  OK  [0m] Stopped Regular background program processing daemon.
[[0;32m  OK  [0m] Stopped iptables_exporter.service.
[[0;32m  OK  [0m] Stopped Accounts Service.
[[0;32m  OK  [0m] Stopped Deferred execution scheduler.
[[0;32m  OK  [0m] Stopped AWS X-Ray Daemon.
[[0;32m  OK  [0m] Stopped node_exporter.service.
[[0;32m  OK  [0m] Stopped OpenBSD Secure Shell server.
[[0;32m  OK  [0m] Stopped Unattended Upgrades Shutdown.
[[0;32m  OK  [0m] Stopped Authenticate and Authorize Users to Run Privileged Tasks.
[[0;32m  OK  [0m] Stopped Getty on tty1.
[[0;32m  OK  [0m] Stopped Serial Getty on ttyS0.
[[0;32m  OK  [0m] Stopped filebeat.service.
[[0;32m  OK  [0m] Stopped journalbeat.service.
[[0;32m  OK  [0m] Stopped OpenTelemetry Collector Contrib.
[[0;32m  OK  [0m] Stopped spooler.service.
[[0;32m  OK  [0m] Stopped sensorcore_endpoint.service.
[[0;32m  OK  [0m] Stopped ffmpeg_server.service.
[[0;32m  OK  [0m] Stopped sensorcore_registration.service.
[[0;32m  OK  [0m] Stopped Snap Daemon.
[[0;32m  OK  [0m] Stopped D-Bus System Message Bus.
[[0;32m  OK  [0m] Stopped LSB: Set up cgroupfs mounts..
[[0;32m  OK  [0m] Stopped LSB: automatic crash report generation.
[[0;32m  OK  [0m] Stopped FUSE filesystem for LXC.
[[0;32m  OK  [0m] Stopped LSB: Start/stop sysstat's sadc.
[[0;32m  OK  [0m] Stopped LSB: daemon to balance interrupts for SMP systems.
[[0;32m  OK  [0m] Stopped LSB: Set the CPU Frequency Scaling governor to "ondemand".
[[0;32m  OK  [0m] Stopped LSB: Record successful boot for GRUB.
[[0;32m  OK  [0m] Stopped LXD - container startup/shutdown.
[[0;32m  OK  [0m] Stopped LSB: MD monitoring daemon.
         Unmounting Mount unit for amazon-ssm-agent, revision 2648...
         Unmounting Mount unit for core, revision 9804...
         Unmounting Mount unit for core18, revision 1880...
         Unmounting Mount unit for core, revision 9665...
         Unmounting Mount unit for core18, revision 1885...
[[0;32m  OK  [0m] Removed slice system-serial\x2dgetty.slice.
[[0;32m  OK  [0m] Removed slice system-getty.slice.
[[0;32m  OK  [0m] Stopped /etc/rc.local Compatibility.
         Stopping Permit User Sessions...
         Stopping Login Service...
[[0;32m  OK  [0m] Stopped rn-misc-exporter.service.
[[0;32m  OK  [0m] Stopped Permit User Sessions.
[[0;32m  OK  [0m] Unmounted Mount unit for core18, revision 1880.
[[0;32m  OK  [0m] Unmounted /var/lib/docker/container...fb1e5683dcea1385d309/mounts/shm.
[[0;32m  OK  [0m] Unmounted Mount unit for amazon-ssm-agent, revision 2648.
[[0;32m  OK  [0m] Unmounted /var/lib/docker/overlay2/...1b30f0a915cec76c43f2aa0f/merged.
[[0;32m  OK  [0m] Unmounted Mount unit for core18, revision 1885.
[[0;32m  OK  [0m] Unmounted Mount unit for core, revision 9804.
[[0;32m  OK  [0m] Stopped Login Service.
[[0;32m  OK  [0m] Unmounted Mount unit for core, revision 9665.
[[0;32m  OK  [0m] Stopped target User and Group Name Lookups.
[[0;32m  OK  [0m] Stopped rabbitmq-exporter.service.
[[0;32m  OK  [0m] Unmounted /var/lib/docker/container...09464762e6c42294844a/mounts/shm.
[[0;32m  OK  [0m] Unmounted /var/lib/docker/overlay2/...05e4a02a3acb174eec87c875/merged.
[[0;32m  OK  [0m] Stopped rabbitmq.service.
         Stopping Docker Application Container Engine...
[[0;32m  OK  [0m] Stopped Docker Application Container Engine.
[[0;32m  OK  [0m] Stopped chrony-wait.service.
         Stopping LSB: Controls chronyd NTP time daemon...
[[0;32m  OK  [0m] Stopped LSB: Controls chronyd NTP time daemon.
[[0;32m  OK  [0m] Stopped target System Time Synchronized.
[[0;32m  OK  [0m] Stopped target Remote File Systems.
[[0;32m  OK  [0m] Stopped target Remote File Systems (Pre).
         Stopping Login to default iSCSI targets...
[[0;32m  OK  [0m] Stopped Login to default iSCSI targets.
         Stopping iSCSI initiator daemon (iscsid)...
[[0;32m  OK  [0m] Stopped iSCSI initiator daemon (iscsid).
[[0;32m  OK  [0m] Stopped target Network is Online.
[[0;32m  OK  [0m] Stopped Service for snap applicatio...azon-ssm-agent.amazon-ssm-agent.
[[0;32m  OK  [0m] Stopped target Network.
         Unmounting Mount unit for amazon-ssm-agent, revision 2758...
[[0;32m  OK  [0m] Stopped target Basic System.
[[0;32m  OK  [0m] Stopped target Slices.
[[0;32m  OK  [0m] Removed slice User and Session Slice.
[[0;32m  OK  [0m] Stopped target Paths.
[[0;32m  OK  [0m] Stopped Forward Password Requests to Wall Directory Watch.
[[0;32m  OK  [0m] Stopped Dispatch Password Requests to Console Directory Watch.
[[0;32m  OK  [0m] Stopped ACPI Events Check.
[[0;32m  OK  [0m] Stopped target Sockets.
[[0;32m  OK  [0m] Closed Docker Socket for the API.
[[0;32m  OK  [0m] Closed UUID daemon activation socket.
[[0;32m  OK  [0m] Closed ACPID Listen Socket.
[[0;32m  OK  [0m] Closed Socket activation for snappy daemon.
[[0;32m  OK  [0m] Closed D-Bus System Message Bus Socket.
[[0;32m  OK  [0m] Closed LXD - unix socket.
[[0;32m  OK  [0m] Stopped target System Initialization.
         Stopping Update UTMP about System Boot/Shutdown...
[[0;32m  OK  [0m] Stopped target Encrypted Volumes.
         Stopping Load/Save Random Seed...
[[0;32m  OK  [0m] Stopped target Swap.
[[0;32m  OK  [0m] Stopped Initial cloud-init job (metadata service crawler).
         Stopping Raise network interfaces...
[[0;32m  OK  [0m] Stopped Load/Save Random Seed.
[[0;32m  OK  [0m] Stopped Update UTMP about System Boot/Shutdown.
[[0;32m  OK  [0m] Stopped Create Volatile Files and Directories.
[[0;32m  OK  [0m] Unmounted Mount unit for amazon-ssm-agent, revision 2758.
[[0;32m  OK  [0m] Stopped Raise network interfaces.
[[0;32m  OK  [0m] Stopped target Network (Pre).
[[0;32m  OK  [0m] Stopped Initial cloud-init job (pre-networking).
[[0;32m  OK  [0m] Stopped Apply Kernel Variables.
[[0;32m  OK  [0m] Stopped target Local File Systems.
         Unmounting /run/docker/netns/default...
[[0;32m  OK  [0m] Stopped Load Kernel Modules.
[[0;32m  OK  [0m] Unmounted /run/docker/netns/default.
[[0;32m  OK  [0m] Reached target Unmount All Filesystems.
[[0;32m  OK  [0m] Stopped target Local File Systems (Pre).
[[0;32m  OK  [0m] Stopped Remount Root and Kernel File Systems.
[[0;32m  OK  [0m] Stopped Create Static Device Nodes in /dev.
[[0;32m  OK  [0m] Reached target Shutdown.
[2106492.792350] reboot: Restarting system

danr-amz commented 4 years ago

We tried to reproduce this issue on Ubuntu Server 16.04, with kernel 4.4.233-0404233-generic obtained from here, and SSM Agent version 2.3.1569.0. However, the instance boots back up as expected after rebooting from either the local shell or from a RunCommand document.

Can you please provide us with the following information to help us reproduce the issue?

How was the reboot triggered? If you triggered the reboot through AWS Systems Manager (e.g. from a RunCommand document), can you please provide the document?
Do you know the exact kernel version? Which kernel modules are installed?

Thanks.

nouse commented 4 years ago

The kernel version is linux-image-4.4.0-187-generic 4.4.0-187.217, I will close this issue first and try to reproduce it.

aws / amazon-ssm-agent

Generic Linux kernel incompatibile with v2.3.1569.0? #302