ApplyAnsiblePlaybooks: aws:runShellScript fails to run playbook on AL2 due to yum lock

[snorlaX-sleeps]

Hi - unsure if this is the correct location to post this, related to the install prerequisite stage of the AWS-ApplyAnsiblePlaybooks document.

The SSM document AWS-ApplyAnsiblePlaybooks will try to install OS specific packages when the parameter InstallDependencies == True in the association. We have been getting errors occasionally about yum locks on Amazon Linux 2 instances - what I believe is happening is another process is currently using yum and is holding the lock. The document then fails to execute and the whole run is aborted.

We use this SSM document to configure our instances on first boot and having it fail before the execution of the playbook (essentially silently) is not going to work long term for 1000's of instances.

We have encountered this before with custom Ubuntu images when the apt lock is held. There should be a wait or retry check for the lock before attempting to install packages, rather than failure.

Error:

Installing and or updating required tools: Ansible, wget unzip ....
Existing lock /var/run/yum.pid: another copy is running as pid 5775.
Another app is currently holding the yum lock; waiting for it to exit...
  The other application is: yum
    Memory : 185 M RSS (401 MB VSZ)
    Started: Tue Feb 22 20:54:34 2022 - 00:02 ago
    State  : Running, pid: x
Traceback (most recent call last):
  File "/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 375, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 281, in main
    return_code = base.doTransaction()
  File "/usr/share/yum-cli/cli.py", line 816, in doTransaction
    resultobject = self.runTransaction(cb=cb)
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1851, in runTransaction
    self.skipped_packages, rpmdb_problems, cmdline)
  File "/usr/lib/python2.7/site-packages/yum/history.py", line 939, in beg
    yum.misc.getloginuid()))
  File "/usr/lib/python2.7/site-packages/yum/sqlutils.py", line 168, in executeSQLQmark
    return cursor.execute(query, params)
sqlite3.OperationalError: database is locked
Error: Nothing to do
/var/lib/amazon/ssm/<instance_id>/document/orchestration/<execution_id>/runShellScript/_script.sh: line 49: ansible-playbook: command not found
failed to run commands: exit status 127

Code Section:

f  [[ "True" == True ]] ; then
   echo "Installing and or updating required tools: Ansible, wget unzip ...." >&2
   if [ -f  "/etc/system-release" ] ; then
     if cat /etc/system-release|grep -i 'Amazon Linux release 2' ; then
       sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
       sudo yum install -y ansible
       sudo yum -y install unzip
     elif cat /etc/system-release|grep -i 'Amazon Linux AMI' ; then
       sudo pip install ansible --upgrade
       sudo yum -y install unzip
     elif cat /etc/system-release|grep -i 'Red Hat Enterprise Linux' ; then
       sudo yum -y install python3-pip
       sudo pip3 install ansible
       sudo yum -y install unzip
     else
       echo "There was a problem installing or updating the required tools for the document. You can review the log files to help you correct the problem." >&2
       exit 1
     fi
   elif cat /etc/issue|grep -i Ubuntu ; then
     UBUNTU_VERSION=$(cat /etc/issue | grep -i ubuntu | awk '{print $2}' |  awk -F'.' '{print $1}')
     sudo apt-get update
     if [ $(($UBUNTU_VERSION > 18)) == 1 ]; then
       sudo DEBIAN_FRONTEND=noninteractive apt-get install python3-pip -y
       sudo pip3 install ansible --upgrade
     else
       sudo DEBIAN_FRONTEND=noninteractive apt-get install python-pip -y
       sudo pip install ansible --upgrade
     fi
     sudo DEBIAN_FRONTEND=noninteractive apt-get install unzip -y
   else
     echo "There was a problem installing or updating the required tools for the document. You can review the log files to help you correct the problem." >&2
     exit 1
   fi
fi