eclipse-kuksa / kuksa-android-sdk

Apache License 2.0
2 stars 1 forks source link

Fix flakey SDK test cases #85

Closed Chrylo closed 6 months ago

Chrylo commented 6 months ago

Describe the bug The issue #45 introduced the configuration cache. All flakey tests have become even more flakey with this change.

To Reproduce Steps to reproduce the behavior:

  1. `./gradlew test -Dkotest.tags="!CustomDatabroker"

Expected behavior Consistent successful test builds

wba2hi commented 6 months ago

Issue here seems to be, that due to the introduction of the configuration-cache the 'testDebugUnitTest' and 'testReleaseUnitTest' tasks are called and executed in parallel. Meaning the same tests might run at the same time against the same databroker. This may lead to conflicts with our current assertions (e.g. on how many callbacks should be received, or what value a vssPath should have after an update)

When using the --no-configuration-cache flag less tasks are executed in parallel and the above mentioned tasks are ran consecutively so mostly no conflicts happen. In rare cases there is still a test which is failing. These tests should be identified and adapted in a more robust way.

Possible solutions: