search

plus3it / spel

STIG-Partitioned Enterprise Linux (spel)
Other
95 stars 61 forks source link

SPEL AMI no longer contains cfn-signal #546

Closed eseglem closed 2 years ago

eseglem commented 2 years ago

The latest (2022.01.1) SPEL AMI no longer contains cfn-signal. It seems to be traces back to https://github.com/plus3it/spel/pull/542 where more things have been removed from the base image.

Based on the PR it indicated that the pre-installed packages on the image are intended to align with the packages available on Amazon Linux 2, but several files are missing from /opt/aws/bin/ :

I also looked through the files on the new yum repo https://spel-packages.cloudarmor.io/list.html#repo/el7/packages/noarch/ however I cannot seem to find one that includes them.

On an Amazon Linux 2 rpm -qf tells me that they were installed via the aws-cfn-bootstrap-2.0-9.amzn2.noarch package. And ec2-utils-1.2-45.amzn2.noarch for the ec2-metadata one. And there is no package named aws-cfn-bootstrap in any of the repos on the SPEL image. Looking back through the PR, it looks like that package was added to the manifest files, however it is not installed. Probably because none of the repos have that package?

Would it be possible to get this missing utils added back to the image? We utilize cfn-signal for all of our Cloudformation deployments to signal success of failure, and are unable to use the latest image.

lorengordon commented 2 years ago

Hi @eseglem! Thanks for the detailed issue report. The cfn-bootstrap utility was especially problematic to package for el7 and el8, but we really did want to continue providing the utilities in the spel images. So what we did is fall back to the tar.gz archive install that is supported by aws. It is installed as a python package with the python3 runtime. You can see the listing in the manifest, here:

https://github.com/plus3it/spel/blob/b2f94ac9e6001752d6445d5c4c75a43165e0a93f/manifests/spel-minimal-centos-7-hvm.manifest.txt#L4

The install is managed by this function, in amigen7:

https://github.com/plus3it/amigen7/blob/786a7b4fa48eb01f93754f2015972305378e7528/AWSutils.sh#L386-L422

With this installation mechanism, the scripts will not be in /opt/aws/bin/, but they should be in /usr/local/bin, I think. Or almost certainly, in the python3 scripts directory.

A further option for this particular case, and one I've seen many people switch to as maintenance and feature releases of the cfn-bootstrap utilities has been pretty woeful, is switch to using the aws cli for signaling... https://docs.aws.amazon.com/cli/latest/reference/cloudformation/signal-resource.html

That said, we can probably find some way to put the scripts in the prior location...

eseglem commented 2 years ago

Thanks for the thorough response. That does explain a lot of our confusion on this. I had seen that line in the manifest, but didn't realize it was doing special logic for that package. I just wasn't seeing where I expected it to be.

I can confirm that there is a copy of them in /usrl/local/bin now that I looked there. So we can probably put some logic in our script to look both places if necessary. Or, perhaps just using aws cloudformation signal-resource would be the better route. As that should work on any version of the AMI. We will discuss it.

We have nightly builds that automatically select the latest SPEL AMI and deploy on it, so the script moving broke them. But that is also the point of having the nightly build do that. We immediately find out when a tweak like this happens and are able to investigate. Rather than getting surprised later down the line. In the mean time we set most of them to be back on 2021.12 while we investigated the cause and look for fixes.

I do think it would be a nice convenience to have symlinks from /opt/aws/bin for consistency with Amazon Linux 2. Seems like its already being done for cfn-hup, just not the rest. Now that we know the alternative though, I don't think its super critical either.

lorengordon commented 2 years ago

Yeah, doing it for cfn-hup was mandatory due to the way the service file is written. It hardcodes that path. But it didn't occur to me to do it for the other utilities as well, as I tend to just manage my PATH in my scripts and I always add /usr/local/bin...