alibaba / nacos

an easy-to-use dynamic service discovery, configuration and service management platform for building cloud native applications.
https://nacos.io
Apache License 2.0
30.36k stars 12.86k forks source link

Flaky Behavior in NewMetadataSelector Tests #12884

Closed DaniellaHubble closed 4 days ago

DaniellaHubble commented 4 days ago

Describe the bug The tests testNewMetadataSelector and testNewMetadataSelector2 in the client module at com.alibaba.nacos.client.naming.selector.NamingSelectorFactoryTest are flaky because they rely on non-deterministic collections (like HashMap), which do not guarantee consistent iteration order. This randomness causes intermittent test failures, leading to false negatives in automated testing.

Expected behavior The tests should consistently pass with reliable results, regardless of the underlying collection order.

Actually behavior The tests fail intermittently due to non-deterministic collection ordering, leading to inconsistent results. This flakiness was detected using NonDex, a testing tool that identifies flaky tests caused by non-deterministic behaviors, such as random or unordered execution.

How to Reproduce Steps to reproduce the behavior:

  1. Navigate to the nacos directory
  2. Run the the above mentioned tests (with nondex) using the following commands mvn -pl client edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=com.alibaba.nacos.client.naming.selector.NamingSelectorFactoryTest#testNewMetadataSelector and mvn -pl client edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=com.alibaba.nacos.client.naming.selector.NamingSelectorFactoryTest#testNewMetadataSelector2
3. See assertion error in the NonDex log [ERROR] NamingSelectorFactoryTest.testNewMetadataSelector:114 expected: com.alibaba.nacos.api.naming.pojo.Instance@5a9d6f02 but was: com.alibaba.nacos.api.naming.pojo.Instance@5a9d6f02 [INFO] [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

Desktop: