search

SkynetLabs / ansible-playbooks

SkynetLabs Ansible Playbooks
6 stars 5 forks source link

draft: run skynet-js tests remotely #313

Closed kwypchlo closed 2 years ago

kwypchlo commented 2 years ago

This is a draft proof of concept

Currently integration tests are run from deploy machine against the server that is being deployed. This limits us in terms of the performance on integration tests that run in parallel and make up most of the deploy time. Instead, we can run those tests on the servers that are being deployed. Docker will clean up any leftovers after the tests are finished. It also dramatically simplifies the job, we could actually add it to "playbooks/tasks/portal-execute-commands.yml" instead.

Running integration tests on 4 servers in parallel.

Before (removed serial: 1 from this playbook to run like it runs during deploy):

Saturday 21 May 2022  13:38:05 +0000 (0:00:03.173)       0:05:01.528 **********
===============================================================================
Run integration tests locally against the portal ---------------------- 283.96s
Remove test container --------------------------------------------------- 3.17s
Checkout skynet-js repo ------------------------------------------------- 2.21s
Install Docker and docker-compose SDKs for Python ----------------------- 2.15s
Download skynet-js tags via Github API ---------------------------------- 2.07s
Get highest valid skynet-js semver version ------------------------------ 1.40s
Create devops/logs directory -------------------------------------------- 1.37s
Get Skynet API key from .env file --------------------------------------- 1.13s
Get Accounts limit access from .env file -------------------------------- 1.07s
Include running integration tests --------------------------------------- 0.52s
Include getting skynet-js repository latest valid semver version -------- 0.41s
Set skynet-js versions -------------------------------------------------- 0.25s
Fail if you are targeting all dev and prod webportals ------------------- 0.25s
Include preparing portal prerequisities --------------------------------- 0.25s
Fail if you are targeting more then max_hosts --------------------------- 0.22s
Fail if Skynet API is not defined --------------------------------------- 0.22s
Fail if no valid skynet-js semver release was found --------------------- 0.22s
Log latest skynet-js version -------------------------------------------- 0.20s
Set skynet-js latest version -------------------------------------------- 0.20s
SUCCESS: Command finished successfully

After:

Saturday 21 May 2022  13:18:20 +0000 (0:01:56.743)       0:02:01.573 **********
===============================================================================
Run integration tests ------------------------------------------------- 116.74s
Install Docker and docker-compose SDKs for Python ----------------------- 2.41s
Create devops/logs directory -------------------------------------------- 1.25s
Include running integration tests --------------------------------------- 0.27s
Include preparing portal prerequisities --------------------------------- 0.23s
Fail if you are targeting more then max_hosts --------------------------- 0.22s
Fail if you are targeting all dev and prod webportals ------------------- 0.22s
SUCCESS: Command finished successfully

We could even further improve the execution time if we build a node docker image with skynet-js already installed and ready to run tests.

MSevey commented 2 years ago

Nice I like this.

We could even further improve the execution time if we build a node docker image with skynet-js already installed and ready to run tests.

I think this is reasonable. Then we just add it to the docker compose file and all ansible has to do it target it with a simple command. That also allows us to run integration tests on a cron job nightly if we wanted.

kwypchlo commented 2 years ago

While this is true that tests are running much faster if they are run from a target machine, they were intentionally executed from a different machine so that they test portal end-to-end.

If e.g. firewall is misconfigured, we get passing tests but unreachable portal.

That is still the case, the docker image is totally isolated and uses an external domain to access the server so it tests it in the same way running from different server would.