Add Integration Tests - Githubissues

arm4b commented 4 years ago

There is a CI check which mimicks quick smoke tests for the Docker deployment to verify it somewhat works: https://github.com/StackStorm/st2-docker/blob/02c7d2f0a046dbc3cb2b6aac788ce065c1fd6bde/.circleci/config.yml#L33-L38

Better way would be following the solution in existing stackstorm-ha repo and adopt proper BATS integration tests: https://github.com/StackStorm/stackstorm-ha/blob/master/tests/st2tests.sh that would check system aliveness.

Implement these tests so the proper docker-compose entry point to run would be as simple as: docker-compose run tests

minsis commented 3 years ago

Is this something to run inside of a container?

arm4b commented 3 years ago

Right, something like tests container with the st2 CLI and BATS framework installed to run the st2 checks in an automated way.

blag commented 3 years ago

We already run end-to-end tests with BATS for StackStorm/st2:

It might be useful to figure out which of those tests also cleanly work against an st2-docker instance and reuse those tests in integrations tests for this repo.

If we have that working, I would add that config to the StackStorm/st2 CI, so changes made there don't break st2-docker.

arm4b commented 3 years ago

In this context it would be nice adding a couple of quick infra smoke tests verifying the st2 components/APIs are running, content was loaded and the system works OK in the Docker environment (https://github.com/StackStorm/stackstorm-ha/blob/master/tests/st2tests.sh).

minsis commented 3 years ago

Just kinda poking around at ideas here and maybe get some feedback.

Doing some research there's no real clean way of adding something like st2tests or test to the docker-compose file without it starting up with the rest of the stack. Doing some testing I created a 2nd docker-compose file called st2test.yaml which you can call with docker-compose -f st2test.yaml up.

Its an extra flag to add to docker-compose, but I'm not seeing any other way to do something as simple as docker-compose run test

version: '3'

services:
  st2test:
    image: ${ST2_IMAGE_REPO:-stackstorm/}st2actionrunner:${ST2_VERSION:-latest}
    environment:
      ST2_AUTH_URL: ${ST2_AUTH_URL:-http://st2auth:9100/}
      ST2_API_URL: ${ST2_API_URL:-http://st2api:9101/}
      ST2_STREAM_URL: ${ST2_STREAM_URL:-http://st2stream:9102/}
      ST2WEB_HTTPS: ${ST2WEB_HTTPS:-0}
    command: /st2test.sh
    volumes:
      - ./scripts/st2test.sh:/st2test.sh:ro
    networks:
      - st2-docker_private

networks:
  st2-docker_private:
    external: true

Thoughts?

P.S. If y'all aren't planning to add some kind of specialized test container to st2-dockerfile that includes BATS, then a dockerfile will need to be created and docker-compose will need to run the dockerfile instead of pulling an image from your repo.

arm4b commented 3 years ago

That sounds good to me, so we could use the additional -f st2test.yaml for CI/CD only :+1:

For the testing, we can use a sidecar container (https://medium.com/ncr-edinburgh/docker-tips-tricks-516b9ba41aa2#daf6) in docker-compose that will copy BATS framework to the st2tests container. Similar to https://github.com/StackStorm/stackstorm-ha/blob/master/templates/tests/st2tests-pod.yaml#L17-L24 This way we'll skip creating another Dockerfile.

minsis commented 3 years ago

Need some guidance here. When loading bats into st2tests:/tools/bats-helpers from /opt/bats, none of the bats stuff that the st2test.sh file references is there

load "${BATS_HELPERS_DIR}/bats-support/load.bash"
load "${BATS_HELPERS_DIR}/bats-assert/load.bash"
load "${BATS_HELPERS_DIR}/bats-file/load.bash"

Here is what gets copied over

/tools # ls -al /tools/bats-helpers
total 12
drwxr-xr-x 3 root root 4096 Apr 15 23:06 .
drwxr-xr-x 3 root root 4096 Apr 15 23:06 ..
drwxr-xr-x 3 root root 4096 Apr 15 23:06 bats

/tools # ls -al /tools/bats-helpers/bats
total 32
drwxr-xr-x 3 root root 4096 Apr 15 23:06 .
drwxr-xr-x 3 root root 4096 Apr 15 23:06 ..
-rw-r--r-- 1 root root 1058 Apr 15 23:06 LICENSE
-rw-r--r-- 1 root root 9719 Apr 15 23:06 README.md
drwxr-xr-x 2 root root 4096 Apr 15 23:06 libexec
-rw-r--r-- 1 root root 1673 Apr 15 23:06 package.json

In fact none of the ./bats-*/load.bash files even exist. So are these coming from somewhere else that I'm not seeing?

blag commented 3 years ago

Unfortunately, we use different methods to install bats on different systems when we setup the end-to-end tests:

CentOS/RHEL 7: Install from bats package
CentOS/RHEL 8+: Install bats using npm (yeah, really, and yes, it's weird)
Ubuntu: Install bats from GitHub

I hope, but I'm not certain, that it won't matter for this, as long as bats in in the PATH.

Our our normal end-to-end tests load bats-assert and bats-support, which they pull from st2tests/test_helpers.

Ideally we would use the exact same versions as in the repo, so it might be good to have some logic to copy them straight out of st2tests, or at least figure out the git tags and install the same versions as in the master branch.

Once they are accessible someplace, I think you should be able to either set or extend the BATS_HELPERS_DIR environment variable to wherever they are (as in: path/to/test_helpers) and it should work from there.

minsis commented 3 years ago

@blag Thank you for that. After understanding it a bit I dug a bit deeper into the BATS image that's used for stackstorm-ha and found that the developer also included the same helper libraries.

I have a BATS test working now, with a few of caveats:

Since this repo doesn't load RBAC by default, the test included fails, so instead of removing it I just commented it
The tests that require a login uses the default login that is advertised. Since its included in stackstorm-ha I'm assuming its okay to include it here as well. Plus its in the docs, etc so I guess there isn't a concern for it(?). I've also added it as an env variable so it can be overridden if desired.
I wanted to better contain the tests files so instead of running -f st2test.yaml its now -f tests/st2test.yaml. This also lets you to cd tests/ and run it directly. It also solves the issue of the other containers being in the st2-docker namespace which was causing conflicts with using a second docker-compose file. I also have a feature idea on this to allow users to include their own tests so they can run BATS tests on anything they have developed (going to add this as a separate issue after this is working). Overall I just feel like this is cleaner.
I don't have a huge amount of experience with CI/CD, specifically circleci or github actions. Once the PR goes in I may need some guidance on figuring that out

~ st2-docker (master) $ docker-compose -f tests/st2tests.yaml up
Creating network "tests_default" with the default driver
Creating volume "tests_tools" with default driver
Creating tests_st2test-tools_1 ... done
Creating tests_st2test_1       ... done
Attaching to tests_st2test-tools_1, tests_st2test_1
st2test-tools_1  | 1..0
tests_st2test-tools_1 exited with code 0
st2test_1        | 1..9
st2test_1        | ok 1 st2 version deployed and python env are as expected
st2test_1        | ok 2 ST2_AUTH_URL service endpoint is accessible and working
st2test_1        | ok 3 ST2_API_URL service endpoint is accessible and working
st2test_1        | ok 4 ST2_STREAM_URL service endpoint is accessible and working
st2test_1        | ok 5 st2 user can log in with auth credentials
st2test_1        | ok 6 st2 core pack is installed and loaded
st2test_1        | ok 7 can execute simple st2 action 'core.local'
st2test_1        | ok 8 st2 chatops core rule is loaded
st2test_1        | ok 9 st2 key/value operations are functional
tests_st2test_1 exited with code 0

~ st2-docker (master) $ docker-compose -f tests/st2tests.yaml down -v
Removing tests_st2test_1       ... done
Removing tests_st2test-tools_1 ... done
Removing network tests_default
Network st2-docker_private is external, skipping
Removing volume tests_tools

StackStorm / st2-docker

Add Integration Tests #197