search

hawtio / hawtio-next

Next generation Hawtio UI console
https://hawt.io
Apache License 2.0
7 stars 22 forks source link

E2E - Tests seem flaky because retry isn't counted #893

Closed tadayosi closed 5 months ago

tadayosi commented 5 months ago

E2E has become flaky recently. While the straightforward cause should be migration from CRACO to Webpack, if you look at the test results it appears that test retries are not counted.

The following result seems to show that 3 out of 4 runs passed, so shouldn't this be regarded as the test eventually passed?

https://github.com/hawtio/hawtio-next/actions/runs/8795104020/job/24135649636

Error:  Failures: 
Error:  Attributes and values.Example #1.2
[INFO]   Run 1: PASS
Error:    Run 2: The collection with text elements: [AllowUseOriginalMessage, false, ApplicationContextClassName, org.springframework.boot.loader.launch.LaunchedClassLoader, CamelDescription, null, CamelId, SampleCamel, CamelVersion, 4.4.0, ClassResolver, org.apache.camel.impl.engine.DefaultClassResolver, DeltaProcessingTime, 0, ExchangesCompleted, 12, ExchangesFailed, 0, ExchangesInflight, 0, ExchangesTotal, 12, ExternalRedeliveries, 0, FailuresHandled, 0, FirstExchangeCompletedExchangeId, 5D7E7B9B7D9EEEF-0000000000000000, FirstExchangeCompletedTimestamp, 2024-04-23T05:08:05Z, FirstExchangeFailureExchangeId, null, FirstExchangeFailureTimestamp, null, GlobalOptions, null, HeadersMapFactoryClassName, org.apache.camel.impl.engine.DefaultHeadersMapFactory, IdleSince, 57954, LastExchangeCompletedExchangeId, 5D7E7B9B7D9EEEF-000000000000000B, LastExchangeCompletedTimestamp, 2024-04-23T05:09:00Z, LastExchangeCreatedTimestamp, 2024-04-23T05:09:00Z, LastExchangeFailureExchangeId, null, LastExchangeFailureTimestamp, null, LastProcessingTime, 0, Load01, , Load05, , Load15, , LogMask, false, ManagementName, SampleCamel, ManagementStatisticsLevel, Default, MaxProcessingTime, 10, MeanProcessingTime, 1, MessageHistory, true, MinProcessingTime, 0, PackageScanClassResolver, org.apache.camel.spring.boot.FatJarPackageScanClassResolver, Redeliveries, 0, ResetTimestamp, 2024-04-23T05:08:04Z, ShutdownNowOnTimeout, true, StartedRoutes, 0, StartTimestamp, 2024-04-23T05:08:04Z, State, Suspended, StatisticsEnabled, true, Throughput, 0.00, Timeout, 45, TimeUnit, SECONDS, TotalProcessingTime, 12, TotalRoutes, 2, Tracing, false, Uptime, 1m52s, UptimeMillis, 112962, UseBreadcrumb, false, UseDataType, false, UseMDCLogging, false]
should contain all of the following text elements: [State, Started]
but could not find these elements: [Started]
Collection: table tbody tr td
Screenshot: file:/hawtio-test-suite/tests/hawtio-test-suite/build/reports/tests/1713848999092.2.png
Page source: file:/hawtio-test-suite/tests/hawtio-test-suite/build/reports/tests/1713848999092.2.html
Timeout: 20 s.
[INFO]   Run 3: PASS
[INFO]   Run 4: PASS
tadayosi commented 5 months ago

@mmuzikar Could you take a look at this please?

mmuzikar commented 5 months ago

Should be fixed by https://github.com/hawtio/hawtio/pull/3410, I did some experiments, the different passes were not configured properly and thus not counted.