Continuous Integration Pipeline

EliasGabrielsson commented 5 years ago

When starting thinking of this I came up with a concept of a two level testing which includes four types of test (below). The two levels are hardware based test on all supported platforms (pine64,RPi,x86 Debian), and cloud based test using Travis and therefor indirectly AWS EC2 instances.

The cloud based testing is invoked by Travis on PRs. Travis calls test written with BATS within a docker environment which enable nice setup/teardown and higher degree of reproducibility. How many test we can run in a docker container and dependency therein need to be experimented with. One interesting thought is that if we can do a full installation of openhabian and then do a snapshot of that system which we base further testing on. The BATS test could be filtered with regex, used an category header below.

The hardware test is pretty much the same as the cloud test but on real hardware with added specific hardware tests (Eg. serial port etc.).

To enable this to work in the long run we need to separate GUI code and install scripts more strict. Eg. install_routine_InfluxDB() and install_InfluxDB(ipaddress, port, user) or similar.

Bash Automated Testing System

1. Unit Tests `unit-<name>`

These test does not alter the system and is isolated to a specific function or version. Probably least common as a lot of code configure, setup and install software.

2. Installation Verification `installation-<name>`

This is a suite of tests run directly after an normal installation is complete. This can be used both for testing and post installation verification for end-users. These tests shall not alter the system. All these test cases shall have the following condition:

if [ ! -f "/opt/openHABian-install-successful" ]; then skip; fi

3. Destructive Verification Tests `destructive-<name>`

These test installs new functionality and therefor destruct and change the current system. Typical usecase are testing of optional packages or a specific configuration of a baseline package. The installation function is called within the test-case itself eg.

@test "baseline-influxdb" {
  install_X_feature(parameter)
  result=getResult() // or similar
  [ "$result" ]
}

4. Shellcheck

Checks general formatting of the bash code.

EliasGabrielsson commented 5 years ago

Started elaboration in this private branch: https://github.com/EliasGabrielsson/openhabian/tree/java-tar

Trying to do an installation within docker which maybe easily could be used for teardown and setup between test cases. Everything is done within Travis.

Also tried out the test framework https://github.com/bats-core/bats-core

ThomDietrich commented 5 years ago

Thanks for kicking off efforts!! Docker would also be my way to go here. I did not use bats before but it seems quite capable. Good choice.

Could you explain your approach? I think we need to test functions individually (incl. a recurrent call + update) as well as the whole unattended installation procedure in a whole.

EliasGabrielsson commented 5 years ago

@ThomDietrich Some thoughts written down in header post. There are a lot of room for improvement!

openhab / openhabian