finos / FDC3-conformance-framework

A framework for testing whether desktop containers implement the FDC3 standard
Apache License 2.0
14 stars 15 forks source link

Basic tests can be flaky in a CI system #241

Closed kriswest closed 1 year ago

kriswest commented 1 year ago

We've had intermittent trouble with some of the basic tests on lower-performance or lower number of CPU machines we use in our CI system for testing.

The same tests never fail on higher-performance/higher number of CPU machines. We believe this may be due to a race in the test definition and a slightly different order of execution resulting from a lower number of CPUs. The addition a short wait before the test is confirmed failed will resolve - which is a valid change as there are no specific timing requirements in FDC3 for the things being tested.