search

OpenLiberty / open-liberty

Open Liberty is a highly composable, fast to start, dynamic application server runtime environment
https://openliberty.io
Eclipse Public License 2.0
1.16k stars 597 forks source link

Issues building Open Liberty *locally* #15801

Open tevans78 opened 3 years ago

tevans78 commented 3 years ago

This is a catch-all issue to record problems developers have seen when building OL locally. e.g.

./gradlew assemble should work every time. When it fails, it should fail consistently; running a second time should not change the result. Under normal circumstances, it should not be nescesary to run any kind of clean commmand before running assemble.

The issue should be used to record such failures so that we can try to track down their causes. This is for local build issues only

hutchig commented 3 years ago

Trying parallel out locally while building FATs, I set these vars in my ~/gradle.startup.properties.

buildResultUUID=test
buildLabel=test
is.release=true

Then I hit the failure:

* Where:
Script '/Users/erfritz/workspace/was-new/open-liberty/dev/wlp-gradle/subprojects/fat.gradle' line: 146

* What went wrong:
Execution failed for task ':com.ibm.ws.transaction.core_fat.10:copyFeatureBundles'.
> java.io.FileNotFoundException: /Users/erfritz/workspace/was-new/open-liberty/dev/com.ibm.ws.transaction.core_fat.10/build/libs/buildfiles (No such file or directory)

* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 20s

jar produces build/libs/buildfiles and is running after copyFeatureBundles. Changing mustRunAfter assemble to dependsOn assemble in copyFeatureBundles fixes that issue.

hutchig commented 3 years ago

On today's integration branch 537 git clean -dxf 538 ./gradlew cnf:initialize && ./gradlew assemble

Task :com.ibm.ws.logging:globalize FAILED FAILURE: Build failed with an exception.

  • Where: Script '/Users/hutchig/libertyGit/open-liberty/dev/wlp-gradle/subprojects/tasks.gradle' line: 41
  • What went wrong: Execution failed for task ':com.ibm.ws.logging:globalize'. java.lang.NoClassDefFoundError: org/apache/commons/lang/BooleanUtils

just resubmitting ./gradlew assemble then worked (in the past I would have attempted to do more 'cleaning' which is going in the wrong direction from what is needed (currently)).

hutchig commented 3 years ago

https://ibm-cloud.slack.com/archives/C31DW78RG/p1612433064120000 Apologies if this link does not work for you but it basically says ", this is one of those cases where you can run ./gradlew --stop and be able to rerun it and it will work."

jhanders34 commented 3 years ago

./gradlew assemble does a cnf:initialize so it should not be needed to call cnf:initialize if you are going to do assemble right afterward.

cbridgha commented 3 years ago

Thanks for adding in scenarios - but I think we should reword this issue as a catch all, or open separate issues.

I'll add what I have heard lately, in addition to comments here: 1) Issues with the running daemon caching - required to --stop to recycle VM 2) Problems with running in Parallel - this has worked on and off over the years - BND was the main culprit until last year. 3) Performance issues rerunning assemble - the gradle caching should be helping us, but not all gradle tasks have source/output properly defined so they can be "skipped" on rerun 4) Performance of gradle clean - Clean always goes through the gradle configure tasks, which works out the dependency graph across 1000's of tasks - This could be short circuited to simply delete the folders we know 5) Any tricks should be "baked in" to existing core tasks. 6) Our goal should be never requiring a clean once the entire workspace is built.

Joseph-Cass commented 3 years ago

I've just had 6 different ./gradlew releaseNeeded executions fail. It started with the following:

> Task :com.ibm.ws.org.apache.cxf.cxf.api.2.6.2:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':com.ibm.ws.org.apache.cxf.cxf.api.2.6.2:compileJava'.
> Compilation failed; see the compiler error output for details.

...

BUILD FAILED in 4m 54s
993 actionable tasks: 333 executed, 660 up-to-date

Followed by 4 Java heap space errors, which each took around 20 minutes, and looked like:

> Task :com.ibm.ws.config_fat:jar FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':com.ibm.ws.config_fat:jar'.
> Java heap space

...

BUILD FAILED in 22m 16s
5848 actionable tasks: 3240 executed, 2608 up-to-date

failing on each of the following:

So I tried to do a bit of reading about heap spaces and gradle builds, and found that:

  1. It looks like you can set the start and max heap size by doing the following (from here): export _JAVA_OPTIONS="-Xms1g -Xmx2g"
  2. Our builds already set the values mentioned here: org.gradle.jvmargs=-Xmx2560M -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8 in https://github.com/OpenLiberty/open-liberty/blob/integration/dev/gradle.properties

So I tried setting _JAVA_OPTIONS and after 2 hours got:

> Task :cnf:createGradleBootstrap
Generating gradle bootstrap

FAILURE: Build failed with an exception.

* What went wrong:
Could not receive a message from the daemon.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

And now I give up :(

From origin/integration (commit fbb909be8f9e754b6c0ea995163e07d0d7072c51)

jhanders34 commented 3 years ago

Recently the CTS builds started failing and it was narrowed down to using a new version of gradle. We moved from 6.4 to 6.8 in the automated builds. In order to get around this I have done two different things. For a stop gap we are using 6.4 again with the CTS builds, but I also have tried by updating dev/gradle.properties to use a 4 GB heap in order for things to work. I am working on making a PR to change the heap setting to something greater than the current 2.5 GB, but wanting to find what the right number should be. 4 GB probably is overkill. Trying 3 GB now and will see how that works.

@Joseph-Cass what java version are you using to build. That could be a factor as well. You can update gradle.properties to see if updating the heap setting would be of benefit to you.

@fritze2 Are you aware of new Gradle 6.8 heap usage that wasn't present in previous versions of gradle?

Joseph-Cass commented 3 years ago

I tried setting org.gradle.jvmargs=-Xmx4g -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8 in dev/gradle.properties (and also killed the gradle daemon with ./gradlew --stop) and after 2 hours 2 minutes had a successful build 😅

Java version: 

Josephs-MacBook-Pro:dev josephcass$ java --version
openjdk 13.0.2 2020-01-14
OpenJDK Runtime Environment AdoptOpenJDK (build 13.0.2+8)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.18.0, JRE 13 Mac OS X amd64-64-Bit 20200118_155 (JIT enabled, AOT enabled)
OpenJ9   - 6968c18d7
OMR      - 7a1b0239a
JCL      - 9c5a41bd5f based on jdk-13.0.2+8)
fritze2 commented 3 years ago

No heap size issues that I know of. The list of known issues for 6.8 is https://github.com/gradle/gradle/labels/affects-version%3A6.8 The performance bugs there don't seem to apply to us from what I can tell. It would be valuable to have JVM performance metrics when making those deliveries. As I keep doing this, I will update the wiki process to record that data - https://github.com/OpenLiberty/open-liberty/wiki/Updating-Gradle-and-Bnd-Version

I have also delivered a PR containing changes to enable parallel builds again and reduce the amount of tasks that are configured or executed by Gradle unnecessarily. I've tested the PR on my Mac and a Windows machine. https://github.com/OpenLiberty/open-liberty/pull/15863

hutchig commented 3 years ago

I used to hate the cxf area the most now it is springboot.support:

./gradlew assemble after clean: FAILURE: Build failed with an exception.

rerun Execution failed for task ':com.ibm.ws.springboot.support.version22.test.app:bootJar'.

Main class name has not been configured and it could not be resolved

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 2m 38s 3262 actionable tasks: 139 executed, 1 from cache, 3122 up-to-date ~/libertyGit/open-liberty/dev

$ ./gradlew com.ibm.ws.springboot.support.version22.test.app:bootJar

Task :com.ibm.ws.springboot.support.version22.test.app:compileJava warning: dev :No translation found for macro: artifactory.upload.token warning: dev :No translation found for macro: artifactory.upload.server warning: dev :No translation found for macro: artifactory.upload.user

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 21s

Then the same waste of time for ...support.version23

tevans78 commented 3 years ago

./gradlew clean assemble failed after 5 mins with

> Task :com.ibm.ws.org.apache.cxf.cxf.api.2.6.2:compileJava FAILED
/Users/tevans/Liberty/openLibertyGit/open-liberty/dev/com.ibm.ws.org.apache.cxf.cxf.api.2.6.2/src/org/apache/cxf/staxutils/W3CDOMStreamWriter.java:54: error: cannot find symbol
        document = DOMUtils.newDocument();
                           ^
  symbol:   method newDocument()
  location: class DOMUtils

./gradlew com.ibm.ws.org.apache.cxf.cxf.api.2.6.2:build then passed (17s) ./gradlew assemble finally passed after another 14m 35s

jhanders34 commented 3 years ago

Many of the issues in this issue, if not all, may be resolved by a PR I put in this weekend: #16308. The change disables the gradle cache which will make things build a little slower, but at least they will not have the issues we see here.

jhanders34 commented 3 years ago

I may have been wrong. I just did another build and it failed still. So I am trying again to run using the --no-daemon option when I run. That is how they run on the build systems which could explain why we do not see these issues on the build systems.

jhanders34 commented 3 years ago

That didn't work also. Still getting issues with building

jhanders34 commented 3 years ago

From slack conversations, it looks like it is advantageous to look at updating the gradle version to 6.8.3 from 6.8. Some of these failures could have been introduced by the move to the 6.8 version earlier this year. I have tried with 6.8.3 and did not see the problems that I was seeing over and over with 6.8. It does perform slower than 6.8 with the caching disabled. 6.8.3 may have fixes for the caching problem as well. I ran with and without caching successfully. Usually @fritze2 updates gradle versions which can take some time with running builds and making sure things are formatted correctly. Ethan's updates for better incremental build being put back in may also help reduce the issues with 6.8.3 being worse performing as well.

Joseph-Cass commented 3 years ago

I'm still running into heap space issues when doing ./gradlew clean assemble with the default dev/gradle.properties settings:

> Task :com.ibm.ws.security.oauth.oidc_fat.common:jar FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':com.ibm.ws.security.oauth.oidc_fat.common:jar'.
> Java heap space

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 23m 2s
3032 actionable tasks: 3032 executed

As I mentioned earlier, increasing the max heap size in dev/gradle.properties seems to help:

I tried setting org.gradle.jvmargs=-Xmx4g in dev/gradle.properties and had a successful build

Should we consider increasing this by default? It seems like it could be just another "got'cha" that people could run into. Though i'm not sure why it seems like I'm the only one running into this...