search

stationgroup / ansible-experiments

⚠️ [Moved to r21.io] Ansible experimentation
https://gitlab.com/stationgroup/ansible-experiments
MIT License
2 stars 2 forks source link

Clean up AMI and related snapshots #2

Closed reelsense closed 6 years ago

reelsense commented 6 years ago

I need an Ansible Role/Playbook that can remove old AMI's and the related snapshots that were created using this ec2_ami_module with machines tagged MakeImage=true. I will probably execute this playbook from a scheduled cron job.

ℹ️ Maybe this method I'm imagining above isn't the correct method. Maybe it would be safer to create a Playbook that adds something like a DestroyImage=true tag. Then an additional removal Role that only removes things that are tagged DestroyImage=true. I think this sounds safer as I write this...

Need quote and your input on this also.

x-post: https://github.com/stationgroup/ansible-experiments/pull/8

srgvg commented 6 years ago

I clearly recalled having commented on this issue, and now it seems that is lost, or was not posted.

Most important part of this, is I need a way to test scripts with real AWS AMI info of your infrastructure. Do you see a way how this can happen? At least I'd need an example of return data of the ec2_ami_module

reelsense commented 6 years ago

I’ll setup a testing account that you can start up instances on.

-- PGP:5EC1494DC59141C0

On Apr 29, 2018, at 04:17, Serge van Ginderachter notifications@github.com wrote:

I clearly recalled having commented on this issue, and now it seems that is lost, or was not posted.

Most important part of this, is I need a way to test scripts with real AWS AMI info of your infrastructure. Do you see a way how this can happen? At least I'd need an example of return data of the ec2_ami_module

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

reelsense commented 6 years ago

I emailed the credentials for this experiment and setup a server in the Frankfurt region for you with the correct tags.

If you email a public key I can add you to the authorized_keys file or you can create an EC2 instance and AMI with the System Administrator IAM policy your account has. This way I can pay for any EC2 expenses incurred during this.

srgvg commented 6 years ago

Do you have specific AMI's you manage, and a procedure or script when you make snapshots? Or is there a way to quickly simulate having "old" AMI's and snapshots that should be deleted?

reelsense commented 6 years ago

I'll reply out of band regarding that question.

reelsense commented 6 years ago

There seems to be a very particular issue that I only hit in production.

fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'ebs'\n\nThe error appears to have been in '/Users/USER/github/public-org-stationgroup/ansible-experiments/cleanup_ami_snapshots/cleanup-ami-snapshots.yaml': line 72, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n    - name: Cleanup AMI snapshots\n      ^ here\n"}

I've reproduced it using the test AMI playbook that you made and made a screen recording.

VIDEO LINK

https://www.youtube.com/watch?v=V-1W1o4otrI

Can you add what is needed so that I don't error out under the conditions in the video. It seems like a true value followed by an empty value causes it.

srgvg commented 6 years ago

Can you confirm me the exact ansible version you use? (just to be sure)

reelsense commented 6 years ago 
ansible 2.5.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/USER/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
reelsense commented 6 years ago

The above is in production.

This is the version in the video:

ansible 2.5.4
  config file = None
  configured module search path = [u'/Users/USER/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.5.4/libexec/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.15 (default, May  1 2018, 16:44:08) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)]
reelsense commented 6 years ago

If solving this issue is more trouble than it's worth, let me know.

It's a real scenario; But maybe it's easier for the playbook to print a message that there is a DestroyImage key with an empty value and fail. And then I would just manually go through and find the missing value and remove the key.

vincentvdk commented 6 years ago

Hi, I'm helping @srgvg on this. Could you provide the following information: sudo pip show boto sudo pip show boto3 aws --version

reelsense commented 6 years ago

sudo pip show boto

---
Metadata-Version: 2.0
Name: boto
Version: 2.48.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
Installer: pip
License: MIT
Location: /home/USER/.local/lib/python2.7/site-packages
Requires: 
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  License :: OSI Approved :: MIT License
  Operating System :: OS Independent
  Topic :: Internet
  Programming Language :: Python :: 2
  Programming Language :: Python :: 2.6
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4

sudo pip show boto3

---
Metadata-Version: 2.0
Name: boto3
Version: 1.7.15
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: UNKNOWN
Installer: pip
License: Apache License 2.0
Location: /home/USER/.local/lib/python2.7/site-packages
Requires: jmespath, s3transfer, botocore
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  Natural Language :: English
  License :: OSI Approved :: Apache Software License
  Programming Language :: Python
  Programming Language :: Python :: 2.6
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4
  Programming Language :: Python :: 3.5
  Programming Language :: Python :: 3.6

aws --version

aws-cli/1.15.4 Python/2.7.12 Linux/4.4.0-104-generic botocore/1.10.15
vincentvdk commented 6 years ago

Could you update these and try to reproduce? sudo pip install --upgrade botocore sudo pip install --upgrade boto3 sudo pip install --upgrade boto

reelsense commented 6 years ago

Nothing changed from what I showed in the video.

sudo pip show boto

---
Metadata-Version: 2.0
Name: boto
Version: 2.48.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
Installer: pip
License: MIT
Location: /home/USER/.local/lib/python2.7/site-packages
Requires: 
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  License :: OSI Approved :: MIT License
  Operating System :: OS Independent
  Topic :: Internet
  Programming Language :: Python :: 2
  Programming Language :: Python :: 2.6
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4

sudo pip show boto3

---
Metadata-Version: 2.0
Name: boto3
Version: 1.7.38
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: UNKNOWN
Installer: pip
License: Apache License 2.0
Location: /home/USER/.local/lib/python2.7/site-packages
Requires: s3transfer, jmespath, botocore
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  Natural Language :: English
  License :: OSI Approved :: Apache Software License
  Programming Language :: Python
  Programming Language :: Python :: 2.6
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4
  Programming Language :: Python :: 3.5
  Programming Language :: Python :: 3.6

aws --version

aws-cli/1.15.38 Python/2.7.12 Linux/4.4.0-104-generic botocore/1.10.38
vincentvdk commented 6 years ago

Somehow I cannot reproduce this. video capture: https://youtu.be/Fxm0RSXRLrY

reelsense commented 6 years ago

Strange... You copied my method exactly in the video. I can reproduce it on 2 separate AWS accounts, on 2 different OS's, in 2 different regions. One with IAM Roles for permissions, and one with IAM access keys (Administrator Access).

Thank you for your help. I'll close this and email @srgvg now.

U-496024757E7019832