Backport CI improvements

tmetzke commented 1 year ago

Acceptance Criteria (Required on creation)

Appropriate CI stability improvements made with https://github.com/camunda/camunda-bpm-platform/issues/2782 are backported to the supported maintenance versions.

Hints

Removal of JDK 8 and standalone web apps stages cannot be backported (we still support this with <7.20)

Links

Breakdown

### Pull Requests
- [ ] https://github.com/camunda/camunda-bpm-platform-maintenance/pull/1030
- [ ] https://github.com/camunda/camunda-bpm-platform-maintenance/pull/1031
- [ ] https://github.com/camunda/camunda-bpm-platform-maintenance/pull/1032
- [ ] https://github.com/camunda/camunda-bpm-platform-ee-maintenance/pull/422
- [ ] https://github.com/camunda/camunda-bpm-platform-ee-maintenance/pull/423
- [ ] https://github.com/camunda/camunda-bpm-platform-ee-maintenance/pull/424

tmetzke commented 1 year ago

Measures to backport:

7.19
7.18
- [x] https://github.com/camunda/camunda-bpm-platform/issues/3428
- [x] https://github.com/camunda/camunda-bpm-platform/issues/3431
- [x] https://github.com/camunda/camunda-bpm-platform/issues/3430
- [x] https://github.com/camunda/camunda-bpm-platform/issues/2598
- [x] https://github.com/camunda/camunda-bpm-platform/issues/2859
- [x] https://github.com/camunda/camunda-bpm-platform/issues/2605
- [x] https://github.com/camunda/camunda-bpm-platform/issues/2904
- [x] ~https://github.com/camunda/camunda-bpm-platform/issues/3535~ - introduces regressions and the problem did not occur in 7.18
- [x] https://github.com/camunda/camunda-bpm-platform/issues/3530
7.17
- [x] https://github.com/camunda/camunda-bpm-platform/issues/3428
- [x] https://github.com/camunda/camunda-bpm-platform/issues/3431
- [x] https://github.com/camunda/camunda-bpm-platform/issues/3430
- [x] https://github.com/camunda/camunda-bpm-platform/issues/2598
- [x] ~https://github.com/camunda/camunda-bpm-platform/issues/2859~ - no WAS Liberty in 7.17
- [x] https://github.com/camunda/camunda-bpm-platform/issues/2605
- [x] https://github.com/camunda/camunda-bpm-platform/issues/2904
- [x] ~https://github.com/camunda/camunda-bpm-platform/issues/3535~ - introduces regressions and the problem did not occur in 7.17
- [x] https://github.com/camunda/camunda-bpm-platform/issues/3530

tmetzke commented 1 year ago

Review handover

All improvements have been backported.
- Most of them could be cherry-picked as is. Some had to be adjusted since the adjusted classes/tests don't exist on the respective branch.
- 7.17 does not support Websphere Liberty, thus the improvement was not backported.
As far as I can see, all remaining CI failures are "leftovers" that also occur in 7.20 occasionally (some of them less often though).
The CE PRs still fail in the EE Main Pipeline since they ran on the main branch, not the respective feature branch. The EE PRs of the respective Camunda versions are all successful.

Follow-up tasks

If the review is successful, please go ahead and merge the PRs yourself. Otherwise, they are going to sit idle until after my FTO (end of July).
If there are minimal changes and you're confident to apply them, go ahead and also merge afterward.
If there are bigger issues, I can look into them after my FTO.
Depending on when exactly the changes are merged, the potential:xyz labels need to be adjusted to the respective patch version labels.

yanavasileva commented 1 year ago

Notes:

I assume adjustments in ProcessEngineRule#starting are not required anymore after #3530 adjustments but they also don't do any harm.
Applied some clean ups:
- for webapps pom in 7.18/7.17 and cfgAuthorizationCheckRevokesAlways profile
- comment for BatchMigrationTest in test-old-engine and 7.19/7.18/7.17
re: #3428 and remove of cxf stage
- in internal-dependencies/pom.xml, a mule-module-cxf dependency is still defined, but I cannot confirm where it's used

yanavasileva commented 1 year ago

TODO:

[x] check CI results after merge.

yanavasileva commented 1 year ago

After merging the changes there are some failure that I would say are random. Sidetrack old-engine stages were interrupted after 1h timeout. Nothing looks related to the applied changes. Therefore I am closing the ticket.

tmetzke commented 1 year ago

@tasso94's comment in https://github.com/camunda/camunda-bpm-platform/issues/3535

I reopened this ticket to investigate why the engine-rest-unit-resteasy stage permanently fails for 7.17 and 7.18.

I marked the builds for reference as "Keep forever":

https://ci-pipeline.cambpm.camunda.cloud/blue/organizations/jenkins/7.17%2Fcambpm-ce%2Fcambpm-main/detail/7.17/142/pipeline/678

https://ci-pipeline.cambpm.camunda.cloud/blue/organizations/jenkins/7.18%2Fcambpm-ce%2Fcambpm-main/detail/7.18/85/pipeline/533

Stdout
``` [2023-07-17T12:40:28.729Z] [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in org.camunda.bpm.engine.rest.standalone.EqualsMapTest [2023-07-17T12:40:28.729Z] [INFO] Running org.camunda.bpm.engine.rest.standalone.HalResourceCacheTest [2023-07-17T12:40:28.830Z] [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.098 s - in org.camunda.bpm.engine.rest.standalone.HalResourceCacheTest [2023-07-17T12:40:28.830Z] [INFO] Running org.camunda.bpm.engine.rest.standalone.NoServletAuthenticationFilterTest [2023-07-17T12:40:32.244Z] [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.318 s - in org.camunda.bpm.engine.rest.standalone.NoServletAuthenticationFilterTest [2023-07-17T12:40:32.245Z] [INFO] Running org.camunda.bpm.engine.rest.standalone.NoServletEmptyBodyFilterTest [2023-07-17T13:43:01.761Z] Sending interrupt signal to process [2023-07-17T13:43:02.758Z] [INFO] [2023-07-17T13:43:02.758Z] [INFO] Results: [2023-07-17T13:43:02.758Z] [INFO] [2023-07-17T13:43:02.758Z] [WARNING] Tests run: 3412, Failures: 0, Errors: 0, Skipped: 2 [2023-07-17T13:43:02.758Z] [INFO] [2023-07-17T13:43:02.758Z] [INFO] ------------------------------------------------------------------------ [2023-07-17T13:43:02.758Z] [INFO] BUILD FAILURE [2023-07-17T13:43:02.758Z] [INFO] ------------------------------------------------------------------------ [2023-07-17T13:43:02.758Z] [INFO] Total time: 01:05 h [2023-07-17T13:43:02.758Z] [INFO] Finished at: 2023-07-17T15:43:02+02:00 [2023-07-17T13:43:03.113Z] [INFO] Final Memory: 38M/885M [2023-07-17T13:43:03.113Z] [INFO] ------------------------------------------------------------------------ [2023-07-17T13:43:03.214Z] [INFO] [jenkins-event-spy] Generated /home/work/workspace/7.17_cambpm-ce_cambpm-main_7.17@tmp/withMavencad9eb03/maven-spy-20230717-143703-7534853967219029836931.log [2023-07-17T13:43:03.214Z] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on project camunda-engine-rest-core: There are test failures. [2023-07-17T13:43:03.214Z] [ERROR] [2023-07-17T13:43:03.214Z] [ERROR] Please refer to /home/work/workspace/7.17_cambpm-ce_cambpm-main_7.17/engine-rest/engine-rest/target/surefire-reports for the individual test results. [2023-07-17T13:43:03.214Z] [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream. [2023-07-17T13:43:03.214Z] [ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [2023-07-17T13:43:03.214Z] [ERROR] Command was /bin/sh -c cd /home/work/workspace/7.17_cambpm-ce_cambpm-main_7.17/engine-rest/engine-rest && /tmp/jdk1.8.0_181/jre/bin/java -Xmx968m -Duser.language=en -Duser.region=US -jar /home/work/workspace/7.17_cambpm-ce_cambpm-main_7.17/engine-rest/engine-rest/target/surefire/surefirebooter1234416723087412712.jar /home/work/workspace/7.17_cambpm-ce_cambpm-main_7.17/engine-rest/engine-rest/target/surefire 2023-07-17T14-37-57_847-jvmRun1 surefire5147940527596234034tmp surefire_01398425970575193132tmp [2023-07-17T13:43:03.214Z] [ERROR] Error occurred in starting fork, check output in log [2023-07-17T13:43:03.214Z] [ERROR] Process Exit Code: 143 [2023-07-17T13:43:03.214Z] [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [2023-07-17T13:43:03.215Z] [ERROR] Command was /bin/sh -c cd /home/work/workspace/7.17_cambpm-ce_cambpm-main_7.17/engine-rest/engine-rest && /tmp/jdk1.8.0_181/jre/bin/java -Xmx968m -Duser.language=en -Duser.region=US -jar /home/work/workspace/7.17_cambpm-ce_cambpm-main_7.17/engine-rest/engine-rest/target/surefire/surefirebooter1234416723087412712.jar /home/work/workspace/7.17_cambpm-ce_cambpm-main_7.17/engine-rest/engine-rest/target/surefire 2023-07-17T14-37-57_847-jvmRun1 surefire5147940527596234034tmp surefire_01398425970575193132tmp [2023-07-17T13:43:03.215Z] [ERROR] Error occurred in starting fork, check output in log [2023-07-17T13:43:03.215Z] [ERROR] Process Exit Code: 143 [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:282) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:245) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216) [2023-07-17T13:43:03.215Z] [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:160) [2023-07-17T13:43:03.215Z] [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [2023-07-17T13:43:03.215Z] [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [2023-07-17T13:43:03.215Z] [ERROR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [2023-07-17T13:43:03.215Z] [ERROR] at java.lang.reflect.Method.invoke(Method.java:498) [2023-07-17T13:43:03.215Z] [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) [2023-07-17T13:43:03.215Z] [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) [2023-07-17T13:43:03.215Z] [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) [2023-07-17T13:43:03.215Z] [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) [2023-07-17T13:43:03.215Z] [ERROR] -> [Help 1] [2023-07-17T13:43:03.215Z] [ERROR] [2023-07-17T13:43:03.215Z] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [2023-07-17T13:43:03.215Z] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [2023-07-17T13:43:03.215Z] [ERROR] [2023-07-17T13:43:03.215Z] [ERROR] For more information about the errors and possible solutions, please read the following articles: [2023-07-17T13:43:03.215Z] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [2023-07-17T13:43:03.319Z] script returned exit code 1 ```

tmetzke commented 1 year ago

Proposal to move on here

For now, we don't adjust REST tests on 7.17 and 7.18.
If we encounter occasional errors in the future on those versions, we can look into this again.

Reasoning

Version 7.17 will be out of support soon anyway.
After 1h of investigation, I couldn't identify why the adjustments don't work on those versions.
The tests seem to "lock up" in a deadlock and not progress further. This could be due to a different server version and locked-up resources that block starting a server, for example. This is pure speculation at this point, however.
Further investigation might not be worth the time if we don't have the initial issue on the maintenance branches (past builds don't show this issue, we have to observe future build failures).

camunda / camunda-bpm-platform