search

gpestana / kapacitor-unit

Testing framework for Kapacitor TICKscripts
MIT License
75 stars 19 forks source link

Tests sometimes FAIL when they should return OK #8

Closed emuh1 closed 7 years ago

emuh1 commented 7 years ago

Tests sometimes FAIL when they should return OK. If the test run again the test succeeds.

Kapacitor is running in a VM with the IP 192.168.59.101

To reproduce the error, run kapacitor multiple times and check the output. Sometimes a test fails, that was successful last time (see Alert weather:: critical)

Console Output:

weefreeman@snuk-wee:~/go/src/github.com/gpestana/kapacitor-unit$ kapacitor-unit -dir sample/ -kapacitor http://192.168.59.101:9092 -tests sample/test_case.yaml 
2017/09/12 11:26:41 TEST Alert weather:: critical (alert_weather.tick) OK
2017/09/12 11:26:41 TEST Alert weather:: critical (alert_weather.tick) OK
2017/09/12 11:26:41 TEST Alert weather:: failure (alert_weather.tick) FAIL
Should have triggered 2 Ok alerts, triggered 0
Should have triggered 1 Warning alerts, triggered 0
Should have triggered 1 Critical alerts, triggered 0
Alerts triggered (ok: 0, warn: 0, crit: 0)

2017/09/12 11:26:41 TEST Alert weather:: invalid - recording_id + data (alert_weather.tick) ERROR: Configuration file cannot define a recording_id and line protocol data input for the same test case

weefreeman@snuk-wee:~/go/src/github.com/gpestana/kapacitor-unit$ kapacitor-unit -dir sample/ -kapacitor http://192.168.59.101:9092 -tests sample/test_case.yaml 
2017/09/12 11:26:43 TEST Alert weather:: critical (alert_weather.tick) OK
2017/09/12 11:26:43 TEST Alert weather:: critical (alert_weather.tick) FAIL
Should have triggered 1 Warning alerts, triggered 0
Alerts triggered (ok: 0, warn: 0, crit: 0)

2017/09/12 11:26:43 TEST Alert weather:: failure (alert_weather.tick) FAIL
Should have triggered 2 Ok alerts, triggered 0
Should have triggered 1 Critical alerts, triggered 0
Alerts triggered (ok: 0, warn: 1, crit: 0)

2017/09/12 11:26:43 TEST Alert weather:: invalid - recording_id + data (alert_weather.tick) ERROR: Configuration file cannot define a recording_id and line protocol data input for the same test case
gpestana commented 7 years ago

Hey @emuh1, thanks for reporting this! I see that you are using the sample/test_case.yaml example. Currently, that test case configuration has some tests that pass and some others that don't pass. I've been using it as a fast way to manually verify is everything is working as expected (see it as a MVP for component tests). Some tests are failing and they are supposed to fail.

It is indeed a bit confusing to keep this test case in master without any documentation. I will probably prioritise issue nr #5 and once we have the component tests, make sure that the test_case.yaml which is used as an example passes all tests!

Thanks again for reporting this though! Would you like to help with #5? :)

gpestana commented 7 years ago

@emuh1 Closing this issue due to explanation above. Feel free to re-open it if any question or help with #5. Thanks!