search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.07k stars 3.8k forks source link

teamcity: automate jepsen testing #12098

Closed knz closed 7 years ago

knz commented 7 years ago

So we need to get two things rolling:

(requested by @tamird / @cuongdo )

jordanlewis commented 7 years ago

How long do the tests take? Do they require multiple actual hosts, or do they work with docker on a single host?

knz commented 7 years ago

There are several tests (at least 5 we're interested in). Each test runs for a minimum of 5 minutes, although Aphyr recommends we run them for 45 minutes. These are cluster tests which need to change the network configuration of the cluster while each test is running, so they require multiple dedicated VMs/hosts.

The way it's commonly used is 1 "master" host then 3-5 VMs to run the tests on.

jordanlewis commented 7 years ago

It will be much easier to automate this using something like Terraform than trying to coordinate multiple TeamCity agents. I do not recommend trying the latter approach!

Can we try to write a Terraform configuration for this setup, similar to how most of the Nightly tests work? That way, TeamCity will simply trigger the Terraform configuration and watch the tests run remotely.

cuongdo commented 7 years ago

Yes, a Terraform config seems best given the need for multiple hosts. Let me know if you need help. The Terraform configs we use for the nightly allocator & load tests are in pkg/acceptance/terraform.

jordanlewis commented 7 years ago

Update:

jordanlewis commented 7 years ago

As a reminder to the script author: