Better protections and guidelines for AWS tests

[mikedlr]

Proposal: Better protections and guidelines for AWS tests

Author: Your Name mikedlr

Date: 2017/04/07

• Status: New
• Proposal type: test design
• Targeted Release: 2.4
• Associated PR: to come
• Estimated time to implement: 3 weeks of discussion

Motivation

make it safer to run

Problems

What problems exist that this proposal will solve?

• the integration tests have not been run much
• new test are arriving which do not fit the current scheme
• different integration tests
• there is a risk of working on the wrong account

Solution proposal

N.B. this is the proposal that came out of discussions on IRC more than my personal proposal. One of the major feelings is that the current situation is non-standard and inconvenient. However,

• use a single dedicated AWS profile name

• set all of the environment variables that might override the profile to blank

• abort if credentials.yml is configured

• clearly document this and other safeguards and make gudelines for people to follow

• enforce some of this in the CI testing.

• ensure that all cloud resources are created with a prefix

• distribute a policy which can be used to limit the integration tests user to the minimum privileges needed

• only the profile will work and it will only work when explicitly configured so the user is much less likely to make a mistake

Dependencies (optional)

There is a partial dependency on the plan to turn on automatic cloud integration tests in shippable.

Testing (optional)

The integration tests should be checked with various environment variables set to invalid values and it should be verified that this does not disrupt them.

Documentation (optional)

There needs to be an explicit document about what to do when writing

Anything else?

This proposal needs to be reviewed and verified by multiple people working in different AWS environments. There is complexity in getting this right because many different configurations exist.

An early (too open but also a bit incomplete) policy exists here https://gist.github.com/michael-dev2rights/77f9b007d06519d85792a872db4b687f A branch with some environment cleanup is starting here. https://github.com/mikedlr/ansible/tree/mdd-integration-testing-clean-env

[gundalow]

Sounds good. We currently have some AWS specific documentation at https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/GUIDELINES.md

That will get converted to RST and be part of the main dev_guide pages in the future, though if for the moment things get added to the current location that will ensure they will get migrated

[mikedlr]

I have now written a set of proposed guidelines for integration tests for AWS

Also I'm attempting to write a role which will partly enforce them.

• all normal test cases would use the AWS_PROFILE environment variable which would be forced to point to the ansible-integration-test profile.
• slightly abnormal test cases could use the environment or direct setting of module parameters to change credentials used
• test cases which need broken credentials should set AWS_PROFILE=""
• test cases which really need AWS_PROFILE unset would have to be in a special dedicated role.

[gundalow]

Testing docs have been ported to RST and can be found at http://docs.ansible.com/ansible/dev_guide/testing_integration.html#cloud-tests

[mikedlr]

@mattclay has started moving the cloud integration tests from the old system to the new one. As he is doing this he's decided to go in the opposite direction from the one @willthames proposed and @s-hertel and I were starting to implement.

Attached (with Matt's permission) IRC chat log between myself and matt (with all other comments / activity filtered out) ansible-new-cloud-int-test-system-discussion.txt

[bcoca]

Since collections aws development has moved out of core, this should move to community or aws specific proposal process.