search

apache / camel-k

Apache Camel K is a lightweight integration platform, born on Kubernetes, with serverless superpowers
https://camel.apache.org/camel-k
Apache License 2.0
848 stars 345 forks source link

Flaky TestKitMaxBuildLimit #5344

Open squakez opened 2 months ago

squakez commented 2 months ago 
❌ TestKitMaxBuildLimit (16m10.71s)
  OLM is not available in the cluster. Fallback to regular installation.
  Camel K installed in namespace test-3521c82f-435c-43b9-9184-e18af3e60f5a  (global mode)
  integration kit "integration-a" created
  integration kit "integration-b" created
  integration kit "integration-c" created
      build_test.go:137: 
          Timed out after 900.001s.
          Expected
              <v1.BuildPhase>: Error
          to equal
              <v1.BuildPhase>: Succeeded

Observed multiple times. Seen lately in https://github.com/apache/camel-k/actions/runs/8620164219/attempts/1

christophd commented 2 months ago

Some logs from the E2E test dump:

dump.go:337:     > {"level":"info","ts":"2024-04-09T18:06:14Z","logger":"camel-k.maven.build","msg":"BUILD FAILURE"}
      dump.go:337:     > {"level":"info","ts":"2024-04-09T18:06:14Z","logger":"camel-k.maven.build","msg":"------------------------------------------------------------------------"}
      dump.go:337:     > {"level":"info","ts":"2024-04-09T18:06:14Z","logger":"camel-k.maven.build","msg":"Total time:  1.589 s (Wall Clock)"}
      dump.go:337:     > {"level":"info","ts":"2024-04-09T18:06:14Z","logger":"camel-k.maven.build","msg":"Finished at: 2024-04-09T18:06:14Z"}
      dump.go:337:     > {"level":"info","ts":"2024-04-09T18:06:14Z","logger":"camel-k.maven.build","msg":"------------------------------------------------------------------------"}
      dump.go:337:     > {"level":"error","ts":"2024-04-09T18:06:14Z","logger":"camel-k.maven.build","msg":"Failed to execute goal io.quarkus:quarkus-maven-plugin:3.8.3:build (build-integration) on project camel-k-integration: Failed to build quarkus application: Failed to bootstrap application in NORMAL mode: Failed to inject extension deployment dependencies for org.apache.camel.k.integration:camel-k-integration:2.4.0-SNAPSHOT: Failed to collect dependencies of org.apache.camel.quarkus:camel-quarkus-timer-deployment:jar:3.8.1: either its POM could not be resolved from the available Maven repositories or the artifact does not have any dependencies while at least a dependency on the runtime artifact org.apache.camel.quarkus:camel-quarkus-timer:jar:3.8.1 is expected -> [Help 1]","stacktrace":"github.com/apache/camel-k/v2/pkg/util/log.Logger.Error\n\tgithub.com/apache/camel-k/v2/pkg/util/log/log.go:81\ngithub.com/apache/camel-k/v2/pkg/util/maven.normalizeLog\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_lo
  g.go:83\ngithub.com/apache/camel-k/v2/pkg/util/maven.MavenLogHandler\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:54\ngithub.com/apache/camel-k/v2/pkg/util.scan\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:76\ngithub.com/apache/camel-k/v2/pkg/util.RunAndLog.func1\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:55\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78"}
      dump.go:337:     > {"level":"error","ts":"2024-04-09T18:06:14Z","logger":"camel-k.maven.build","msg":"","stacktrace":"github.com/apache/camel-k/v2/pkg/util/log.Logger.Error\n\tgithub.com/apache/camel-k/v2/pkg/util/log/log.go:81\ngithub.com/apache/camel-k/v2/pkg/util/maven.normalizeLog\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:83\ngithub.com/apache/camel-k/v2/pkg/util/maven.MavenLogHandler\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:54\ngithub.com/apache/camel-k/v2/pkg/util.scan\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:76\ngithub.com/apache/camel-k/v2/pkg/util.RunAndLog.func1\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:55\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78"}
      dump.go:337:     > {"level":"error","ts":"2024-04-09T18:06:14Z","logger":"camel-k.maven.build","msg":"To see the full stack trace of the errors, re-run Maven with the -e switch.","stacktrace":"github.com/apache/camel-k/v2/pkg/util/log.Logger.Error\n\tgithub.com/apache/camel-k/v2/pkg/util/log/log.go:81\ngithub.com/apache/camel-k/v2/pkg/util/maven.normalizeLog\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:83\ngithub.com/apache/camel-k/v2/pkg/util/maven.MavenLogHandler\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:54\ngithub.com/apache/camel-k/v2/pkg/util.scan\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:76\ngithub.com/apache/camel-k/v2/pkg/util.RunAndLog.func1\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:55\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78"}
      dump.go:337:     > {"level":"error","ts":"2024-04-09T18:06:14Z","logger":"camel-k.maven.build","msg":"Re-run Maven using the -X switch to enable full debug logging.","stacktrace":"github.com/apache/camel-k/v2/pkg/util/log.Logger.Error\n\tgithub.com/apache/camel-k/v2/pkg/util/log/log.go:81\ngithub.com/apache/camel-k/v2/pkg/util/maven.normalizeLog\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:83\ngithub.com/apache/camel-k/v2/pkg/util/maven.MavenLogHandler\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:54\ngithub.com/apache/camel-k/v2/pkg/util.scan\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:76\ngithub.com/apache/camel-k/v2/pkg/util.RunAndLog.func1\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:55\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78"}
      dump.go:337:     > {"level":"error","ts":"2024-04-09T18:06:14Z","logger":"camel-k.maven.build","msg":"","stacktrace":"github.com/apache/camel-k/v2/pkg/util/log.Logger.Error\n\tgithub.com/apache/camel-k/v2/pkg/util/log/log.go:81\ngithub.com/apache/camel-k/v2/pkg/util/maven.normalizeLog\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:83\ngithub.com/apache/camel-k/v2/pkg/util/maven.MavenLogHandler\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:54\ngithub.com/apache/camel-k/v2/pkg/util.scan\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:76\ngithub.com/apache/camel-k/v2/pkg/util.RunAndLog.func1\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:55\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78"}
      dump.go:337:     > {"level":"error","ts":"2024-04-09T18:06:14Z","logger":"camel-k.maven.build","msg":"For more information about the errors and possible solutions, please read the following articles:","stacktrace":"github.com/apache/camel-k/v2/pkg/util/log.Logger.Error\n\tgithub.com/apache/camel-k/v2/pkg/util/log/log.go:81\ngithub.com/apache/camel-k/v2/pkg/util/maven.normalizeLog\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:83\ngithub.com/apache/camel-k/v2/pkg/util/maven.MavenLogHandler\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:54\ngithub.com/apache/camel-k/v2/pkg/util.scan\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:76\ngithub.com/apache/camel-k/v2/pkg/util.RunAndLog.func1\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:55\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78"}
      dump.go:337:     > {"level":"error","ts":"2024-04-09T18:06:14Z","logger":"camel-k.maven.build","msg":"[Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException","stacktrace":"github.com/apache/camel-k/v2/pkg/util/log.Logger.Error\n\tgithub.com/apache/camel-k/v2/pkg/util/log/log.go:81\ngithub.com/apache/camel-k/v2/pkg/util/maven.normalizeLog\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:83\ngithub.com/apache/camel-k/v2/pkg/util/maven.MavenLogHandler\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:54\ngithub.com/apache/camel-k/v2/pkg/util.scan\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:76\ngithub.com/apache/camel-k/v2/pkg/util.RunAndLog.func1\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:55\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78"}
      dump.go:337:     > {"level":"info","ts":"2024-04-09T18:06:14Z","logger":"camel-k.builder","msg":"step failed with error: failure while building project: : exit status 1","step":"github.com/apache/camel-k/v2/pkg/builder/BuildQuarkusRunner","phase":"20","task":"builder"}
      dump.go:337:     > {"level":"info","ts":"2024-04-09T18:06:14Z","logger":"camel-k.controller.build","msg":"Build duration 2.882894572s","request-namespace":"test-4da959ed-78fb-4ebc-bd74-4452ad37aec1","request-name":"integration-b","build-attempt":5,"build-result":"Error","build-duration":2.882894572,"api-version":"camel.apache.org/v1","kind":"Build","ns":"test-4da959ed-78fb-4ebc-bd74-4452ad37aec1","name":"integration-b"}
      dump.go:337:     > {"level":"error","ts":"2024-04-09T18:06:14Z","logger":"camel-k.controller.build","msg":"Build integration-b failed: failure while building project: : exit status 1","request-namespace":"test-4da959ed-78fb-4ebc-bd74-4452ad37aec1","request-name":"integration-b","api-version":"camel.apache.org/v1","kind":"Build","ns":"test-4da959ed-78fb-4ebc-bd74-4452ad37aec1","name":"integration-b","stacktrace":"github.com/apache/camel-k/v2/pkg/util/log.Logger.Errorf\n\tgithub.com/apache/camel-k/v2/pkg/util/log/log.go:66\ngithub.com/apache/camel-k/v2/pkg/controller/build.(*monitorRoutineAction).updateBuildStatus\n\tgithub.com/apache/camel-k/v2/pkg/controller/build/monitor_routine.go:216\ngithub.com/apache/camel-k/v2/pkg/controller/build.(*monitorRoutineAction).runBuild\n\tgithub.com/apache/camel-k/v2/pkg/controller/build/monitor_routine.go:200"}
      dump.go:337:     > {"level":"info","ts":"2024-04-09T18:06:14Z","logger":"camel-k.controller.build","msg":"State transition","request-namespace":"test-4da959ed-78fb-4ebc-bd74-4452ad37aec1","request-name":"integration-b","api-version":"camel.apache.org/v1","kind":"Build","ns":"test-4da959ed-78fb-4ebc-bd74-4452ad37aec1","name":"integration-b","phase-from":"Running","phase-to":"Failed"}
      dump.go:337:     > {"level":"info","ts":"2024-04-09T18:06:14Z","logger":"camel-k.controller.integrationkit","msg":"Invoking action build","request-namespace":"test-4da959ed-78fb-4ebc-bd74-4452ad37aec1","request-name":"integration-b","api-version":"camel.apache.org/v1","kind":"IntegrationKit","ns":"test-4da959ed-78fb-4ebc-bd74-4452ad37aec1","name":"integration-b"}
      dump.go:337:     > {"level":"info","ts":"2024-04-09T18:06:14Z","logger":"camel-k.controller.build","msg":"State transition","request-namespace":"test-4da959ed-78fb-4ebc-bd74-4452ad37aec1","request-name":"integration-b","api-version":"camel.apache.org/v1","kind":"Build","ns":"test-4da959ed-78fb-4ebc-bd74-4452ad37aec1","name":"integration-b","phase-from":"Failed","phase-to":"Error"}
      dump.go:337:     > {"level":"info","ts":"2024-04-09T18:06:14Z","logger":"camel-k.controller.build","msg":"Build error","request-namespace":"test-4da959ed-78fb-4ebc-bd74-4452ad37aec1","request-name":"integration-b","api-version":"camel.apache.org/v1","kind":"Build","ns":"test-4da959ed-78fb-4ebc-bd74-4452ad37aec1","name":"integration-b","reason":"failure while building project: : exit status 1","error-message":"failure while building project: : exit status 1"}
squakez commented 2 months ago

This is interesting. Never seen that though :/

squakez commented 2 months ago

Seen again in https://github.com/apache/camel-k/actions/runs/8645066005/job/23701493911#step:4:14936. This time the error is:

 dump.go:337:     > {"level":"error","ts":"2024-04-11T10:20:37Z","logger":"camel-k.maven.build","msg":"[FATAL] Non-readable POM /etc/maven/m2/io/quarkus/quarkus-bom/3.8.3/quarkus-bom-3.8.3.pom: input contained no data @ /etc/maven/m2/io/quarkus/quarkus-bom/3.8.3/quarkus-bom-3.8.3.pom","stacktrace":"github.com/apache/camel-k/v2/pkg/util/log.Logger.Error\n\tgithub.com/apache/camel-k/v2/pkg/util/log/log.go:81\ngithub.com/apache/camel-k/v2/pkg/util/maven.normalizeLog\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:83\ngithub.com/apache/camel-k/v2/pkg/util/maven.MavenLogHandler\n\tgithub.com/apache/camel-k/v2/pkg/util/maven/maven_log.go:54\ngithub.com/apache/camel-k/v2/pkg/util.scan\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:76\ngithub.com/apache/camel-k/v2/pkg/util.RunAndLog.func1\n\tgithub.com/apache/camel-k/v2/pkg/util/command.go:55\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78"}

I have the feeling this problem is caused by a concurrent access of Maven to the same set of dependencies. AFAIK the multiple access in Maven was already solved, but I start thinking it is really not.

christophd commented 2 months ago

Maybe we should update the Maven version used in the Camel K operator image?

claudio4j commented 2 months ago

Related to this error 

Failed to inject extension deployment dependencies for 
org.apache.camel.k.integration:camel-k-integration:2.4.0-SNAPSHOT: 
Failed to collect dependencies of 
org.apache.camel.quarkus:camel-quarkus-timer-deployment:jar:3.8.1: 
either its POM could not be resolved from the available Maven repositories

While working on the disconnected build, I noticed quarkus-maven-plugin wants to resolve the deployment artifact of the quarkus extension defined in pom.xml, so for whatever reason quarkus-maven-plugin fails to resolve the deployment artifact, then it cannot build the application.