search

telstra / open-kilda

OpenKilda is an open-source OpenFlow controller initially designed for use in a global network with high control-plane latency and a heavy emphasis on latency-centric data path optimisation.
Apache License 2.0
78 stars 53 forks source link

Gradle retry plugin: Unable to handle failed test that has a random unique name per iteration #5633

Open yuliiamir opened 5 months ago

yuliiamir commented 5 months ago

The issue occurs when the parameterized Spock method with a random unique name for each iteration fails. As a result, the Gradle retry plugin can not retrieve old failed test details by new rerun test details.

Issue: 

 FAILURE: Build failed with an exception.
 * What went wrong:
 Execution failed for task ':functional-tests:functionalTest'.
  > The following test methods could not be retried, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues

As a solution, tests that have such kind of naming strategy should be modified.

Example: 

...EnvExtension:-2 - Creating topologies (1)
   EnvExtension:-1 - Starting topology-related HC: lab_id=3
  EnvExtension:-1 - Topology-related HC passed: lab_id=3
...
BaseSpecification:-2 - Booked lab with id 3 for spec HaFlowStatSpec, thread: Thread[Test worker,5,main]. sw: 00:00:00:00:00:00:00:01
...
 INFO  LogParallelSpecsListener:-2 - Running test: HaFlowStatSpec['LOW_PRIORITY', 'HA_FLOW']#System is able to collect latency stats for subflows [subFlow: 11Apr091705_157_oatflour2420_haflow-a, direction: FORWARD, #0]
 ERROR LogParallelSpecsListener:-2 - Specs ran in parallel: [HaFlowStatSpec]
  HaFlowStatSpec > System is able to collect latency stats for subflows ✘ System is able to collect latency stats for subflows [subFlow: 11Apr091705_157_oatflour2420_haflow-a, direction: FORWARD, #0] (1m 1s)

    Condition failed with Exception:

    wait(statsRouterRequestInterval) { if(direction == FORWARD && subFlow.contains("haflow-a")) { assert labService.getLabs().first().labId != 3 } assert flowStats.of(subFlow).get(FLOW_RTT, direction).hasNonZeroValues() }
        at org.openkilda.functionaltests.spec.stats.HaFlowStatSpec.System is able to collect latency stats for subflows(HaFlowStatSpec.groovy:100)
    Caused by: org.openkilda.functionaltests.helpers.Wrappers$WaitTimeoutException: Condition was not satisfied within 60.0 seconds
        at org.openkilda.functionaltests.helpers.Wrappers.wait(Wrappers.groovy:109)
        ... 1 more
    Caused by: Condition not satisfied:
    labService.getLabs().first().labId != 3 // intentional failure to trigger the 1st failure and during rerun the new topology is created and test should pass

    Retry by Gradle due to the failure: 
    HaFlowStatSpec STANDARD_OUT
 EnvExtension:-1 - Starting topology-related HC: lab_id=4
...
 BaseSpecification:-2 - Booked lab with id 4 for spec HaFlowStatSpec, thread: Thread[Test worker,5,main]. sw: 00:00:00:00:00:00:00:01

HaFlowStatSpec > System is able to collect latency stats for subflows > org.openkilda.functionaltests.spec.stats.HaFlowStatSpec.System is able to collect latency stats for subflows [subFlow: 11Apr091946_456_redlentils4861_haflow-b, direction: FORWARD, #1] STANDARD_OUT
    2024-04-11T09:20:22,280 INFO  LogParallelSpecsListener:-2 - Running test: HaFlowStatSpec['LOW_PRIORITY', 'HA_FLOW']#System is able to collect latency stats for subflows [subFlow: 11Apr091946_456_redlentils4861_haflow-b, direction: FORWARD, #1]
  HaFlowStatSpec > System is able to collect latency stats for subflows ✔ System is able to collect latency stats for subflows [subFlow: 11Apr091946_456_redlentils4861_haflow-b, direction: FORWARD, #1] - //test passed after retry

> Task :functional-tests:functionalTest FAILED
:functional-tests:functionalTest (Thread[Execution worker for ':' Thread 4,5,main]) completed. Took 4 mins 20.883 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':functional-tests:functionalTest'.
> The following test methods could not be retried, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues
     org.openkilda.functionaltests.spec.stats.HaFlowStatSpec#System is able to collect latency stats for subflows [subFlow: 11Apr091705_157_oatflour2420_haflow-a, direction: FORWARD, #0]

Failed test: System is able to collect latency stats for subflows [subFlow: 11Apr091705_157_oatflour2420_haflow-a, direction: FORWARD, #0]

Test name after Gradle rerun failed test^: System is able to collect latency stats for subflows [subFlow: 11Apr091946_456_redlentils4861_haflow-b, direction: FORWARD, #1]

yuliiamir commented 4 months ago

The following method naming also caused the same Gradle issue with report generation in the scope of the retry mechanism.

@Unroll("Able to create a second flow if #data.description")
def "Able to create multiple flows on certain combinations of switch-port-vlans"() {

The appropriate changes have been added in the scope https://github.com/telstra/open-kilda/pull/5644