Makeing it easier to test from a developer's laptop

[nmaludy]

Summary

We maintain our own semi-fork of this repo at https://github.com/EncoreTechnologies/ci-stackstorm The reason for this is because this repo isn't great at running from a developers laptop. Specifically it creates directories the ~/ci and ~/virtualenv, along with several other problems (outlined below).

Goals

• I would like to be able to utilize this repo for testing, at some level, so that our code always is testing against the upstream configs
• I would like this repo to be able to be utilized on a developers laptop
• I would like this repo to easily be able to test against multiple packs on the same machine
• I would like this repo to have support for both python2 and python3
• I would like this repo to have support for both Ubuntu and CentOS

Current problems/challenges/etc

Below is a list of things we had to change from these Makefiles and scripts to get the testing to work on a developers laptop:

CI repo

https://github.com/StackStorm-Exchange/ci/blob/master/.circleci/config.yml explicitly sets up the CI repo in ~/ci and and virtualenv in ~/virtualenv. This leads to potential issues when multiple packs are tested on a single system, specifically around the ~/virtualenv. We've had cases where a user forgot to add in a dependency into requirements.txt and because the virtualenv was already setup in ~/virtualenv from another packs' run, we never noticed the problem until we had deployed the pack to our staging environment.

Proposed Changes

• Setup a Makefile that is designed to be placed in a Pack's root directory. This Makefile clones the ci repo and then executes the tests. This Makefile is designed to be kicked off by a user and can coordinate all steps needed to happen in order to run tests. Basically, merge functionality from .circleci/config.yml, .circle/dependencies, .circle/test together into the Makefile.
• Clone the CI repo into ./ci inside the pack's directory instead of ~/ci. Cloning locally means each pack has their own copy when testing against multiple packs on a single machine (preventing file corruption issues).

Dependencies

The dependency script that is run by this CI repo has a couple problems:

• It is not idempotent, so when a developer wants to run their tests a 2nd time, the dependency script errors, specifically around the git clone methods.
• The script hard codes the CI_DIR to be /home/circleci/ci
• The script expects a Debian instance and calls apt-get
• The script creates a ~/virtualenv

Proposed Changes

• Put these dependency resolution steps into a Makefile and have idempotent checks like:

  if [ ! -d "$(ST2_REPO_PATH)" ]; then \
      git clone https://github.com/StackStorm/st2.git --depth 1 --single-branch --branch $(ST2_REPO_BRANCH) $(ST2_REPO_PATH); \
  else \
      cd "$(ST2_REPO_PATH)"; \
      git pull; \
  fi;

• Put the dependencies into ./ci inside the pack's directory instead of ~/ci or /home/circleci/ci
• Put virtualenv into ./ci/virtualenv inside the pack's directory instead of ~/virtualenv
• Don't run any apt-get or yum commands.

packs-resource-register requires MongoDB

Currently the packs-resource-register test requires that a MongoDB instance be running. Unless the user has a docker up and a MongoDB container running, this test is hard to run.

Proposed Changes

• See if we can modify the test to not require MongoDB at all and just validate against known schemas?

[nmaludy]

@Kami @blag if you guys have any thoughts on this, i'd really appreciate it. I'm trying to get away from our own fork of this CI repo and would like to make it easier for pack developers to start testing locally (common problem i've seen on Slack)