search

aws / amazon-ssm-agent

An agent to enable remote management of your EC2 instances, on-premises servers, or virtual machines (VMs).
https://aws.amazon.com/systems-manager/
Apache License 2.0
1.04k stars 323 forks source link

Updating amazon-ssm-agent not working for debian instances #347

Closed pproux closed 3 years ago

pproux commented 3 years ago

Hi,

Following https://github.com/aws/amazon-ssm-agent/issues/233 and https://github.com/aws/amazon-ssm-agent/issues/300

Updating amazon-ssm-agent not working for debian instances.

Error message is : 

Updating amazon-ssm-agent from 2.3.672.0 to latest

Successfully downloaded https://s3.eu-west-1.amazonaws.com/amazon-ssm-eu-west-1/ssm-agent-manifest.json

----------ERROR-------

cannot find the amazon-ssm-agent-windows-amd64.tar.gz information in the Manifest file

OS is 

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Current SSM agent version is 

ii  amazon-ssm-agent                2.3.672.0-1                       amd64        Amazon SSM Agent for managing EC2 Instances using the SSM APIs.
Thor-Bjorgvinsson commented 3 years ago

Hey @pproux, thanks for reporting this issue and taking the time to look for older issues.

Could you paste the output of this command here:

ls /etc

and if the output you pasted above is not from the following file, please post output of this command

cat /etc/os-release

it would be very helpful to look at the logs from the AmazonSSMAgent-update.txt log file in /var/log/amazon/ssm folder Could you paste the output from the following log lines

grep -A 5 "fetching platform details from" /var/log/amazon/ssm/AmazomSSMAgent-update.txt
pproux commented 3 years ago

Hi, thank you for your quick answer !

Here is mine :

ls /etc/

adduser.conf        cron.daily  fstab.old    init.d       logcheck        modules-load.d  passwd-    rc4.d       shadow         terminfo
alternatives        cron.hourly gai.conf     initramfs-tools  login.defs      motd        perl       rc5.d       shadow-            timezone
amazon          cron.monthly    groff        inputrc          logrotate.conf      mtab        php        rc6.d       shells         tmpfiles.d
apache2         crontab     group        iproute2         logrotate.d     nanorc      pm         rcS.d       skel           ucf.conf
apt         cron.weekly group-       issue        machine-id      network     ppp        reportbug.conf  ssh            udev
bash.bashrc     dbus-1      grub.d       issue.net        magic       NetworkManager  profile    resolv.conf     ssl            ufw
bash_completion     debconf.conf    gshadow      kernel       magic.mime      networks    profile.d  rmt         staff-group-for-usr-local  update-motd.d
bash_completion.d   debian_version  gshadow-     ldap         mailcap         newt        protocols  rpc         subgid         vim
bindresvport.blacklist  default     gss      ld.so.cache      mailcap.order   nginx       python     rsyslog.conf    subgid-            wgetrc
binfmt.d        deluser.conf    host.conf    ld.so.conf       manpath.config      nsswitch.conf   python2.7  rsyslog.d       subuid         X11
ca-certificates     dhcp        hostname     ld.so.conf.d     memcached.conf      opt         python3    screenrc        subuid-            xdg
ca-certificates.conf    dpkg        hosts        libaudit.conf    memcached.conf.old  os-release      python3.5  securetty       sudoers            xml
calendar        emacs       hosts.allow  libpaper.d       mime.types      pam.conf    rc0.d      security        sudoers.d          zabbix
chrony          environment hosts.deny   locale.alias     mke2fs.conf     pam.d       rc1.d      selinux         sysctl.conf
cloud           fonts       hosts.orig   locale.gen       modprobe.d      papersize   rc2.d      services        sysctl.d
cron.d          fstab       init         localtime        modules         passwd      rc3.d      sgml        systemd

cat /etc/os-release

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Your last command seems buggy as i can read "amazom" in the provided filename. Furthermore the file "AmazonSSMAgent-update.txt" doesn't exist as well. I ran grep -Ri -A 5 "fetching platform" /var/log/amazon/ssm/* instead, result is empty.

Thor-Bjorgvinsson commented 3 years ago

Thanks for the update, Please follow these steps to get the required logs:

  1. Change the log level of the agent to debug - please follow this guide
  2. Restart the agent (should not be needed but just in case)
  3. Run the update document again
  4. Run the grep you posed in your last comment.
pproux commented 3 years ago

Thanks :)

Here is the output of the grep command

/var/log/amazon/ssm/amazon-ssm-agent.log:2021-01-29 09:25:14 DEBUG [StartupProcessor] fetching platform details from /etc/os-release
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:14 DEBUG [StartupProcessor] Command output &{Debian GNU/Linux 9}
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:14 DEBUG [StartupProcessor] getting platform details
/var/log/amazon/ssm/amazon-ssm-agent.log:2021-01-29 09:25:14 DEBUG [StartupProcessor] fetching platform details from /etc/os-release
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:14 DEBUG [StartupProcessor] Command output &{Debian GNU/Linux 9}
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:14 INFO [StartupProcessor] Write to serial port: Amazon SSM Agent v2.3.672.0 is running
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:14 INFO [StartupProcessor] Write to serial port: OsProductName: Debian GNU/Linux
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:14 INFO [StartupProcessor] Write to serial port: OsVersion: 9
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:14 DEBUG [HealthCheck] Could not fetch FQDN using command /bin/hostname, error exit status 1. Ignoring
--
/var/log/amazon/ssm/amazon-ssm-agent.log:2021-01-29 09:25:14 DEBUG [HealthCheck] fetching platform details from /etc/os-release
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:14 DEBUG [HealthCheck] Command output &{Debian GNU/Linux 9}
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:14 DEBUG [HealthCheck] getting platform details
/var/log/amazon/ssm/amazon-ssm-agent.log:2021-01-29 09:25:14 DEBUG [HealthCheck] fetching platform details from /etc/os-release
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:14 DEBUG [HealthCheck] Command output &{Debian GNU/Linux 9}
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:14 DEBUG [HealthCheck] Calling UpdateInstanceInformation with params{
/var/log/amazon/ssm/amazon-ssm-agent.log-  AgentName: "amazon-ssm-agent",
/var/log/amazon/ssm/amazon-ssm-agent.log-  AgentStatus: "Active",
/var/log/amazon/ssm/amazon-ssm-agent.log-  AgentVersion: "2.3.672.0",
--
/var/log/amazon/ssm/amazon-ssm-agent.log:2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] fetching platform details from /etc/os-release
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] Command output &{Debian GNU/Linux 9}
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] getting platform details
/var/log/amazon/ssm/amazon-ssm-agent.log:2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] fetching platform details from /etc/os-release
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] Command output &{Debian GNU/Linux 9}
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] isSupported flag = true
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] isPluginHandlerFound flag = true
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] isPreconditionEnabled flag = false
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 INFO [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] Running plugin aws:updateSsmAgent
--
/var/log/amazon/ssm/amazon-ssm-agent.log:2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] [pluginName=aws:updateSsmAgent] fetching platform details from /etc/os-release
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] [pluginName=aws:updateSsmAgent] Command output &{Debian GNU/Linux 9}
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] [pluginName=aws:updateSsmAgent] getting platform details
/var/log/amazon/ssm/amazon-ssm-agent.log:2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] [pluginName=aws:updateSsmAgent] fetching platform details from /etc/os-release
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] [pluginName=aws:updateSsmAgent] Command output &{Debian GNU/Linux 9}
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] [pluginName=aws:updateSsmAgent] OrchestrationDir /var/lib/amazon/ssm/i-002f362f6da910a2c/document/orchestration/d6b442de-6407-4bae-afb0-eeebe1e96343/awsupdateSsmAgent 
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] [pluginName=aws:updateSsmAgent] OrchestrationDir /var/lib/amazon/ssm/i-002f362f6da910a2c/document/orchestration/d6b442de-6407-4bae-afb0-eeebe1e96343/awsupdateSsmAgent 
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] [pluginName=aws:updateSsmAgent] https://s3.eu-west-1.amazonaws.com/amazon-ssm-eu-west-1/ssm-agent-manifest.json is valid s3 url
/var/log/amazon/ssm/amazon-ssm-agent.log-2021-01-29 09:25:35 DEBUG [ssm-document-worker] [d6b442de-6407-4bae-afb0-eeebe1e96343] [DataBackend] [pluginName=aws:updateSsmAgent] attempting to download as s3 download /var/log/amazon/ssm/download/update/20792933313b487af758ef11281e2a4d22087705

Here is a part of the /var/log/amazon/ssm/amazon-ssm-agent.log log file

2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] getting platform details
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] fetching platform details from /etc/os-release
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] Command output &{Debian GNU/Linux 9}
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] getting platform details
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] fetching platform details from /etc/os-release
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] Command output &{Debian GNU/Linux 9}
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] OrchestrationDir /var/lib/amazon/ssm/i-002f362f6da9
10a2c/document/orchestration/bd0a1966-6d03-4762-a8c8-88e60ae015de/awsupdateSsmAgent 
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] OrchestrationDir /var/lib/amazon/ssm/i-002f362f6da9
10a2c/document/orchestration/bd0a1966-6d03-4762-a8c8-88e60ae015de/awsupdateSsmAgent 
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] https://s3.eu-west-1.amazonaws.com/amazon-ssm-eu-we
st-1/ssm-agent-manifest.json is valid s3 url
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] attempting to download as s3 download /var/log/amaz
on/ssm/download/update/20792933313b487af758ef11281e2a4d22087705
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] failed to download from s3, AccessDenied: Access De
nied
        status code: 403, request id: 9205C1FB58AAF2B3, host id: GHM7F1m0c9CIBt9AhnlusG5DbFp6o5b8o5ALpPrcxC2uUQXK+9v16h4uKaKmABrxhE6Mfne8ONY=
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] attempting to download as http/https download /var/
log/amazon/ssm/download/update/20792933313b487af758ef11281e2a4d22087705
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] file eTagValue is "de2eb9d97b3389743769bf73c96a4bf6
"
2021-01-29 09:31:40 INFO [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] /var/log/amazon/ssm/download/update/20792933313b487a
f758ef11281e2a4d22087705 with 505062 bytes downloaded
2021-01-29 09:31:40 INFO [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] found package amazon-ssm-agent
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] [pluginName=aws:updateSsmAgent] IOHandler closing all subscribed writers.
2021-01-29 09:31:40 INFO [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] Sending plugin aws:updateSsmAgent completion message
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] plugin: aws:updateSsmAgent done, sending reply message...
2021-01-29 09:31:40 INFO [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] document execution complete
2021-01-29 09:31:40 INFO [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] [DataBackend] sending document complete response...
2021-01-29 09:31:40 DEBUG [ssm-document-worker] [bd0a1966-6d03-4762-a8c8-88e60ae015de] sending datagram: {"version":"1.0","type":"reply","content":"{\"DocumentName\":\"\",\"DocumentVersion\":\"\",\"MessageID\":\"\",\"AssociationID\":\"\",\"PluginResults\":{\"aws:updateSsmAgent\":{\"pluginID\":\"aws:updateSsmAgent\",\"pluginName\":\"aws:updateSsmAgent\",\"status\":\"Failed\",\"code\":1,\"output\":\"Updating amazon-ssm-agent from 2.3.672.0 to latest\\nSuccessfully downloaded https://s3.eu-west-1.amazonaws.com/amazon-ssm-eu-west-1/ssm-agent-manifest.json\\n\\n----------ERROR-------\\ncannot find the amazon-ssm-agent-windows-amd64.tar.gz information in the Manifest file\",\"startDateTime\":\"2021-01-29T09:31:40.646581337Z\",\"endDateTime\":\"2021-01-29T09:31:40.840701741Z\",\"outputS3BucketName\":\"\",\"outputS3KeyPrefix\":\"\",\"stepName\":\"\",\"error\":\"\",\"standardOutput\":\"Updating amazon-ssm-agent from 2.3.672.0 to latest\\nSuccessfully downloaded https://s3.eu-west-1.amazonaws.com/amazon-ssm-eu-west-1/ssm-agent-manifest.json\\n\",\"standardError\":\"cannot find the amazon-ssm-agent-windows-amd64.tar.gz information in the Manifest file\"}},\"Status\":\"InProgress\",\"LastPlugin\":\"aws:updateSsmAgent\",\"NPlugins\":0}"}
gianniLesl commented 3 years ago

Hey @pproux it looks like this issue was fixed in agent version 2.3.871.0 last year. To fix the automated update functionality on your debian instance you'll have to manually install a more recent version.

sudo service amazon-ssm-agent stop
sudo dpkg -r amazon-ssm-agent
mkdir /tmp/ssm 
wget https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/debian_amd64/amazon-ssm-agent.deb -O /tmp/ssm/amazon-ssm-agent.deb 
sudo dpkg -i /tmp/ssm/amazon-ssm-agent.deb

Once you update to latest (or a version greater than or equal to 2.3.871.0) you can utilize the Agent Auto-Update functionality and the AWS-UpdateSSMAgent document.

pproux commented 3 years ago

It works thanks :)