Smoke tests to cover all providers

[prashanth26]

Objective

To have integration/smoke tests

• Across all cloud providers
• Kubernetes versions
• To avoid breakage while adding new features/fixes

Current state of smoke tests

We have integration tests written on bash (only for AWS) where we run a set of smoke tests like

1. Create and delete single machine object
2. Create a machine-deployment with 3 replicas
3. Scale it down to 2
4. Scale it up to 4
5. Rolling update of machines from v1 to v2

Possible solutions moving forward

We have two options moving forward

1. Improve smoke tests to check for all providers and K8s versions by provisioning clusters statically (clusters that are pre-created) /dynamically(on-demand)
2. Mock API calls for different cloud providers to mock providers while running integration tests

Improve smoke tests

Things to consider

• How many MCM would be running?
• Where would MCM be running?
• Different poroviders supporting different k8s versions?
• Control-target only for one provider?
• Clean up - Run a stable controller and remove an partially created resources.

2. Mock API calls

• AWS: mocking available
• Openstack: Two ways, but mocking avaiable
• GCP: Not avaiable
• Azure: Issue/PR was opened and closed

[prashanth26]

After having a follow up discussion on this topic. The following were the notable points in the discussions.

• Going with improving existing smoke tests seems to be the simpler idea as long as we are able to clean up the resources properly.

• Clean up could be provider specific as each provider needs to be handled differently for cleanup.

• Different options to be considered while cleanup are (action items to me) -

  1. Creating (azure) resource-group like tags with each resources created by test MCM
  2. Specific accounts/access keys used to create resources which helps in easier clean-ups
  3. Record-reply tools like Hoverfly could be made use of to track all resources created.

• Multiple levels of integration tests - Where a basic integration (probably that tests only one CP) test would run on every PR. Before every release a more advanced integration test can be run that covers all providers and kubernetes versions.

• The above step could be further simplified by using two different flags to differentiate these test levels. This would require changes in the test flags.

[prashanth26]

After doing a first round of analysis, here are my findings.

Resource groups

• Tagging an instance to an resource group is also something MCM would have to take care of, and is also a possible point of failure on MCM code.
• Further more, all cloud providers do not have support for resource groups. Support for each CP is mentioned below.
• Resource groups support in different cloud providers
  1. Azure - There is implicit support for resource groups.
  2. AWS - There is no implicit support, but you can optionally tag each resource while creation. And later use these tags as filters to create a resource group.
  3. GCP - Also recommends resource groups by proper tagging/labels which can be used for filtering.
  4. Openstack - First looks suggests there there exists no such mechanism.
• Considering the above points, I am against this option.

Account/Access-key specific cleanup

• Cleanup for an account/access-key doesn't exist.
• We could make provider specific monitoring tools like cloudtrail for AWS (similar tools for other providers), to retrace the creation stack and trigger their corresponding deletions.
• However, using tools like this would expect us to develop our own stack of cleanup for each provider seperately which might be time consuming and a tedious task.

[amshuman-kr]

The way I see it, we have three options.

1. Use labels and hope that label-related bugs don't occur too often. This is not as bad as it sounds :-)
2. Use something like hoverfly or mock-server to record cloud provider API calls and then figure out what to cleanup from the recorded requests and responses.
3. Discard integration tests and actually mock the services (perhaps using something like hoverfly or mock-server).

To me, 1 or 3 make sense. 2, not so much. What about everyone else?

[prashanth26]

After having an follow up offline discussion with @amshuman-kr. The following is our plan,

1. Go ahead with option(1) where we continue existing method of using labels to tag VMs backing machines and clean up based on the stale machine objects present in the cluster. If clean-up fails (due to errors with improper labelling in the controller code), the integration test would also fail.
2. Once we have this in place, maybe we can make efforts to go with actual mocking of cloud interfaces as described in option (3).