Karm / mandrel-integration-tests

Integration tests for GraalVM and its Mandrel distribution. Runs Quarkus, Helidon and Micronaut applications and small targeted reproducers. The focus is solely on native-image utility and compilation of Java applications into native executables.
Apache License 2.0
5 stars 3 forks source link

Retry on GOTO URL failure to give more time to the app to start #175

Closed zakkak closed 11 months ago

Karm commented 11 months ago

@zakkak I am not sure. It seems to me we are gradually relaxing this area of the test suite instead of taking a hard look at the root cause.

I understand that we want a sanity/smoke test on random GH CI that is always green unless really broken, and yet: This test suite part is very old. It used to work with older Graals and Quarkuses.

Could we do something like test failure when the re-do is needed? Controlled perhaps by https://github.com/Karm/mandrel-integration-tests/blob/master/testsuite/src/it/java/org/graalvm/tests/integration/utils/Commands.java#L79

Karm commented 11 months ago

My particular issue ATM is running gdb in container that got apparently even more problematic than before:

mvn clean verify -Ptestsuite-builder-image -Dtest=DebugSymbolsTest#debugSymbolsQuarkusContainer 
                           -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-17
                           -Dquarkus.version=3.2.3.Final
jerboaa commented 11 months ago

@Karm As per https://github.com/graalvm/mandrel/pull/545 we have a fairly long delay (~9 secs) until gdb is ready in 23.0.1.1-Final. Better than it used to, but still not great. It suggests we need this retry loop. So I'd suspect that's what you are seeing as well with 3.2 quarkus?

Karm commented 11 months ago

@jerboaa I know about that. I also increased this from 1s to 30s: https://github.com/Karm/mandrel-integration-tests/blob/master/testsuite/src/it/java/org/graalvm/tests/integration/DebugSymbolsTest.java#L411 ...to get at least some output.

And yet, I am unable to match anything, likely due to superfluous bytes? e.g.

Command 'c&' did not match the expected pattern '.*Continuing.*'.
Output was:
"c&
"Continuing.
^running*running,thread-id="all"[Switching to thread 23 (Thread 0x7fffa9afe700 (LWP 50565))](running)

With "Continuing" being there in the buffer, regexp having DOTALL, but perhaps not at the right time?

I am all +1 for making it more robust with re-tries, but the overall test result should be a failure if FAIL_ON_PERF_REGRESSION is true.

The test result message would say it failed due to being too slow, not because it did not process the command...