search

opensearch-project / ml-commons

ml-commons provides a set of common machine learning algorithms, e.g. k-means, or linear regression, to help developers build ML related features within OpenSearch.
Apache License 2.0
96 stars 135 forks source link

[BUG] ML-Commons Plugin build failure due to OpenSearch Core breaking changes #891

Closed b4sjoo closed 1 year ago

b4sjoo commented 1 year ago

What is the bug? ML-Commons Plugin 3.0 build fails due to jackson update in OS core

How can one reproduce the bug?

./gradlew build

 FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'opensearch-ml'.
> java.util.concurrent.ExecutionException: org.gradle.api.GradleException: Failed to create Jar file /home/runner/.gradle/caches/jars-9/bc33e077f18aab4d8905d3690c9e67e4/jackson-core-2.15.0.jar.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org
b4sjoo commented 1 year ago

This is a bug in Gradle - https://github.com/gradle/gradle/issues/24390 Basically we are trying to bump gradle to a version higher than 7.6 gradle migration guide

b4sjoo commented 1 year ago

We need to bump gradle to version 8 because ./gradlew wrapper in gradle 7.6 threw following error messages:

* What went wrong:
An exception occurred applying plugin request [id: 'com.github.johnrengelman.shadow']
> Failed to apply plugin class 'com.github.jengelman.gradle.plugins.shadow.ShadowBasePlugin'.
   > This version of Shadow supports Gradle 8.0+ only. Please upgrade.

following fixing PRs https://github.com/opensearch-project/neural-search/pull/169, https://github.com/opensearch-project/k-NN/pull/899

b4sjoo commented 1 year ago

After bumping ./gradlew wrapper returned error message like 

FAILURE: Build completed with 1 failures.
1: Task failed with an exception.
-----------
* What went wrong:
A problem occurred configuring project ':opensearch-ml-plugin'.
> Failed to notify project evaluation listener.
   > Could not create task ':opensearch-ml-plugin:buildDeb'.
      > You can't map a property that does not exist: propertyName=version
   > Could not create task ':opensearch-ml-plugin:buildRpm'.
      > You can't map a property that does not exist: propertyName=version

buildDeb and buildRpm are two tasks imported by package nebula.ospackage. It also need to be updated so that it can be compatible with gradle 8

b4sjoo commented 1 year ago

./gradlew build failed in test step:

> Task :opensearch-ml-algorithms:compileTestJava FAILED
/local/home/seasonsg/dev/ml-commons/ml-algorithms/src/test/java/org/opensearch/ml/engine/algorithms/anomalylocalization/AnomalyLocalizerImplTests.java:40: error: cannot find symbol
import org.opensearch.common.collect.ImmutableOpenMap;
                                    ^
  symbol:   class ImmutableOpenMap
  location: package org.opensearch.common.collect
1 error

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':opensearch-ml-algorithms:compileTestJava'.
> Compilation failed; see the compiler error output for details.

This is due to OS core breaking changes https://github.com/opensearch-project/OpenSearch/pull/7309

b4sjoo commented 1 year ago

Following this PR https://github.com/opensearch-project/k-NN/pull/885 to fix this problem.

b4sjoo commented 1 year ago

./gradlew build failed in test step:

> Task :opensearch-ml-algorithms:compileTestJava
Note: /local/home/seasonsg/dev/ml-commons/ml-algorithms/src/test/java/org/opensearch/ml/engine/algorithms/anomalylocalization/AnomalyLocalizerImplTests.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :opensearch-ml-algorithms:test
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsOverallDecrease FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsForSearchFailure FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testExecuteFail FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsFilterEntity FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsGivenIndexPattern FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testExecuteInterrupted FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testExecuteSucceed FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsGivenNoAnomaly FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsForInvalidIndexName FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsOverallUnchange FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsForInvalidTimeRange FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsGivenAnomaly FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/djl_cache_2bb316be-484a-439c-a7cb-014ea30f3e25/ml_cache/pytorch/1.12.1-cpu-precxx11-linux-x86_64/libtorch_cpu.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

142 tests completed, 12 failed

> Task :opensearch-ml-algorithms:test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':opensearch-ml-algorithms:test'.
b4sjoo commented 1 year ago

This is due to breaking changes in OS core: https://github.com/opensearch-project/OpenSearch/pull/7301, the customs field in ClusterState class was changed to Map<String, Custom> from ImmutableOpenMap<String, Custom>. Following this PR to fix: https://github.com/opensearch-project/anomaly-detection/pull/886

bbarani commented 1 year ago

Tagging @nknize @reta @saratvemulapalli

b4sjoo commented 1 year ago

The migration from ImmutableOpenMap to java.util.Map finished and can confirm succeed. But now the integTest will fail due to deployModel will fail in RestMLCustomModelActionIT

> Task :opensearch-ml-plugin:integTest

org.opensearch.ml.rest.RestMLCustomModelActionIT > testCustomModelWorkflow FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([39C4FE8636E3D27E]:0)
REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLCustomModelActionIT.testCustomModelWorkflow" -Dtests.seed=39C4FE8636E3D27E -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Africa/Lubumbashi -Druntime.java=17

org.opensearch.ml.rest.RestMLCustomModelActionIT > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
        at __randomizedtesting.SeedInfo.seed([39C4FE8636E3D27E]:0)
REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLCustomModelActionIT.testCustomModelWorkflow" -Dtests.seed=39C4FE8636E3D27E -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Africa/Lubumbashi -Druntime.java=17
reta commented 1 year ago

@bbarani @b4sjoo the plugin has to be updated (sadly)

ylwu-amzn commented 1 year ago

Another breaking change is the string length limit from this PR https://github.com/opensearch-project/OpenSearch/pull/7286#issuecomment-1528016568

By bumping jackson version, we will introduce the string length limitation FasterXML/jackson-core#1001 (comment)

The model content saved as Base64 String in index. As model content string exceeds the string limit, our IT failed. @reta created an https://github.com/opensearch-project/OpenSearch/issues/7549 to support tuning string length limit, before that ready, we have to disable our IT, also neural-search plugin has to disable their IT

b4sjoo commented 1 year ago

The migration from ImmutableOpenMap to java.util.Map finished and can confirm succeed. But now the integTest will fail due to deployModel will fail in RestMLCustomModelActionIT

> Task :opensearch-ml-plugin:integTest

org.opensearch.ml.rest.RestMLCustomModelActionIT > testCustomModelWorkflow FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([39C4FE8636E3D27E]:0)
REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLCustomModelActionIT.testCustomModelWorkflow" -Dtests.seed=39C4FE8636E3D27E -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Africa/Lubumbashi -Druntime.java=17

org.opensearch.ml.rest.RestMLCustomModelActionIT > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
        at __randomizedtesting.SeedInfo.seed([39C4FE8636E3D27E]:0)
REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLCustomModelActionIT.testCustomModelWorkflow" -Dtests.seed=39C4FE8636E3D27E -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Africa/Lubumbashi -Druntime.java=17

As we root causing this issue, we found

  1. The taskid returned in deployModel in RestMLCustomModelActionIT is always a FAILED.
  2. The error message in the response is "error":"{\"Fail to find model chunk {MODEL_ID}_0\"}"

Trying to use this code to continue the RCA.

Update: The PR #884 can reproduce this problem, but definitely not RC because main branch has not merged this.

penghuo commented 1 year ago

Could we have quick fix before RCA? There are 5 version bump PR pending on this https://github.com/opensearch-project/ml-commons/issues/850 and SQL main branch bwc IT pending on this

ylwu-amzn commented 1 year ago

@b4sjoo can we ignore the failed IT/UT to unblock other teams first?

reta commented 1 year ago

@penghuo @ylwu-amzn the fix for Jackson 2.15 content limits went in to main (https://github.com/opensearch-project/OpenSearch/pull/7550)

b4sjoo commented 1 year ago

@b4sjoo can we ignore the failed IT/UT to unblock other teams first?

So we simply delete the problematic IT first?

ylwu-amzn commented 1 year ago

@b4sjoo can we ignore the failed IT/UT to unblock other teams first?

So we simply delete the problematic IT first?

Refer to https://github.com/opensearch-project/ml-commons/blob/194fb595131c1588c4db82213be53df5b2fd9f62/plugin/src/test/java/org/opensearch/ml/rest/RestMLSearchTaskActionIT.java#L22

b4sjoo commented 1 year ago

Problematic IT was ignored and build passed.

b4sjoo commented 1 year ago

Github CI workflow failed due to

> Task :opensearch-ml-client:publishShadowPublicationToMavenLocal FAILED

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
FAILURE: Build failed with an exception.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/8.1.1/userguide/command_line_interface.html#sec:command_line_warnings
* What went wrong:
16 actionable tasks: 10 executed, 6 up-to-date
A problem was found with the configuration of task ':opensearch-ml-client:publishShadowPublicationToMavenLocal' (type 'PublishToMavenLocal').
  - Gradle detected a problem with the following location: '/home/runner/work/ml-commons/ml-commons/client/build/libs/opensearch-ml-client-3.0.0.0-SNAPSHOT.jar'.

    Reason: Task ':opensearch-ml-client:publishShadowPublicationToMavenLocal' uses this output of task ':opensearch-ml-client:jar' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.

    Possible solutions:
      1. Declare task ':opensearch-ml-client:jar' as an input of ':opensearch-ml-client:publishShadowPublicationToMavenLocal'.
      2. Declare an explicit dependency on ':opensearch-ml-client:jar' from ':opensearch-ml-client:publishShadowPublicationToMavenLocal' using Task#dependsOn.
      3. Declare an explicit dependency on ':opensearch-ml-client:jar' from ':opensearch-ml-client:publishShadowPublicationToMavenLocal' using Task#mustRunAfter.

    Please refer to https://docs.gradle.org/8.1.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
b4sjoo commented 1 year ago

Add a task configuration tasks.named("jar").configure { dependsOn("publishShadowPublicationToMavenLocal") } can make the PublishToMavenLocal task succeed.

b4sjoo commented 1 year ago

Github CI workflow now all passed. Please check if problem remains.

b4sjoo commented 1 year ago

PR merged. @penghuo @bbarani Can we check if ml plugin still block any CI?

b4sjoo commented 1 year ago

Neural team and SQL team reported unblock. There are several things we still need to add toTODO, including fixing the IT, BWC, as well as the buildDeb and buildRpm tasks in gradle.build.

b4sjoo commented 1 year ago

Jenkins CI workflow on OS 3.0 failed with message:

FAILURE: Build failed with an exception.

* What went wrong:
A problem was found with the configuration of task ':opensearch-ml-client:publishShadowPublicationToStagingRepository' (type 'PublishToMavenRepository').
  - Gradle detected a problem with the following location: 'C:\Users\Administrator\AppData\Local\Temp\tmpypnhpv6j\ml-commons\client\build\libs\opensearch-ml-client-3.0.0.0.jar'.

    Reason: Task ':opensearch-ml-client:publishShadowPublicationToStagingRepository' uses this output of task ':opensearch-ml-client:jar' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.\

https://build.ci.opensearch.org/job/distribution-build-opensearch/7856/console

We try to fix it with PR #901

b4sjoo commented 1 year ago

Here is an issue board to track most of the breaking changes in OS core in 3.0, please keep an eye on it - https://github.com/opensearch-project/OpenSearch/issues/5910

b4sjoo commented 1 year ago

And also a place to provide feedback - https://github.com/opensearch-project/opensearch-devops/issues/114

b4sjoo commented 1 year ago

Which is due to major refactor in OS core 3.0:

dev/ml-commons/plugin/src/main/java/org/opensearch/ml/cluster/DiscoveryNodeHelper.java:40: error: cannot find symbol
        excludedNodeNames = Strings.commaDelimitedListToSet(ML_COMMONS_EXCLUDE_NODE_NAMES.get(settings));
                                   ^
  symbol:   method commaDelimitedListToSet(String)
  location: class Strings
dev/ml-commons/plugin/src/main/java/org/opensearch/ml/cluster/DiscoveryNodeHelper.java:43: error: cannot find symbol
            .addSettingsUpdateConsumer(ML_COMMONS_EXCLUDE_NODE_NAMES, it -> excludedNodeNames = Strings.commaDelimitedListToSet(it));
                                                                                                       ^
  symbol:   method commaDelimitedListToSet(String)
  location: class Strings

https://github.com/opensearch-project/OpenSearch/pull/7508 However after change the import we got:

dev/ml-commons/common/src/test/java/org/opensearch/ml/common/dataframe/DoubleValueTest.java:54: error: method toString in class Object cannot be applied to given types;
        String jsonStr = Strings.toString(builder);
                                ^
  required: no arguments
  found:    XContentBuilder
  reason: actual and formal argument lists differ in length

Here is a temporary fix method: https://github.com/opensearch-project/security/issues/2782

b4sjoo commented 1 year ago

Created a fix PR https://github.com/opensearch-project/ml-commons/pull/901

I got following integ test error message after migration from org.opensearch.common.Strings to org.opensearch.core.common.Strings

»  java.lang.NoClassDefFoundError: org/opensearch/core/common/Strings
»   at org.opensearch.ml.cluster.DiscoveryNodeHelper.<init>(DiscoveryNodeHelper.java:40)
»   at org.opensearch.ml.plugin.MachineLearningPlugin.createComponents(MachineLearningPlugin.java:233)
»   at org.opensearch.node.Node.lambda$new$16(Node.java:770)
»   at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
»   at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
»   at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
»   at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
»   at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
»   at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
»   at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
»   at org.opensearch.node.Node.<init>(Node.java:784)
»   at org.opensearch.node.Node.<init>(Node.java:377)
»   at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
»   at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
»   at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
»   at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
»   at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
»   at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
»   at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
»   at org.opensearch.cli.Command.main(Command.java:101)
»   at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
»   at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)

The line for DiscoveryNodeHelper.java:40 is

excludedNodeNames = Strings.commaDelimitedListToSet(ML_COMMONS_EXCLUDE_NODE_NAMES.get(settings));

And UnitTests were passed. Any thoughts?

b4sjoo commented 1 year ago

The integ test error is due to docker image not update. Now integ tests were passed. Issue resolved.

b4sjoo commented 1 year ago

Jenkins CI workflow on OS 3.0 failed with message:

FAILURE: Build failed with an exception.

* What went wrong:
A problem was found with the configuration of task ':opensearch-ml-client:publishShadowPublicationToStagingRepository' (type 'PublishToMavenRepository').
  - Gradle detected a problem with the following location: 'C:\Users\Administrator\AppData\Local\Temp\tmpypnhpv6j\ml-commons\client\build\libs\opensearch-ml-client-3.0.0.0.jar'.

    Reason: Task ':opensearch-ml-client:publishShadowPublicationToStagingRepository' uses this output of task ':opensearch-ml-client:jar' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.\

https://build.ci.opensearch.org/job/distribution-build-opensearch/7856/console

We try to fix it with PR #901

We did another fix in #962 due to ml-commons not building when publishing ml-client artifacts.

b4sjoo commented 1 year ago

After opensearch-build team migrate from jdk17 to jdk20, task :opensearch-ml-plugin:delombok failed with following message:

Exception in thread "main" java.lang.NoSuchFieldError: var
    at lombok.delombok.PrettyPrinter.visitForeachLoop(PrettyPrinter.java:1256)
    at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCEnhancedForLoop.accept(JCTree.java:1232)
    at lombok.delombok.PrettyPrinter.print(PrettyPrinter.java:226)
    at lombok.delombok.PrettyPrinter.print(PrettyPrinter.java:240)
    at lombok.delombok.PrettyPrinter.visitBlock(PrettyPrinter.java:1133)
    at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCBlock.accept(JCTree.java:1088)
    at lombok.delombok.PrettyPrinter.print(PrettyPrinter.java:226)
    at lombok.delombok.PrettyPrinter.visitMethodDef(PrettyPrinter.java:857)
    at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCMethodDecl.accept(JCTree.java:912)
    at lombok.delombok.PrettyPrinter.print(PrettyPrinter.java:226)
    at lombok.delombok.PrettyPrinter.printClassMembers(PrettyPrinter.java:617)
    at lombok.delombok.PrettyPrinter.visitClassDef(PrettyPrinter.java:556)
    at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:810)
    at lombok.delombok.PrettyPrinter.print(PrettyPrinter.java:226)
    at lombok.delombok.PrettyPrinter.visitTopLevel(PrettyPrinter.java:482)
    at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:549)
    at lombok.delombok.DelombokResult.print(DelombokResult.java:74)
    at lombok.delombok.Delombok.delombok(Delombok.java:819)
    at lombok.delombok.Delombok.main(Delombok.java:326)
    at lombok.delombok.DelombokApp.runDirectly(DelombokApp.java:176)
    at lombok.delombok.DelombokApp.runApp(DelombokApp.java:47)
    at lombok.core.Main.go(Main.java:130)
    at lombok.core.Main.main(Main.java:43)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.base/java.lang.reflect.Method.invoke(Method.java:578)
    at lombok.launch.Main.main(Main.java:46)

> Task :opensearch-ml-plugin:delombok FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':opensearch-ml-plugin:delombok'.
> Process 'command '/opt/java/openjdk-20/bin/java'' finished with non-zero exit value 1
dhrubo-os commented 1 year ago

I think Exception in thread "main" java.lang.NoSuchFieldError: var issue can be solved by allowing here

But when I try to build ml-commons package with Java 20, 12 tests are failing:

java.lang.instrument.IllegalClassFormatException: Error while instrumenting sun/util/resources/cldr/provider/CLDRLocaleDataMetaInfo with JaCoCo 0.8.8.202204050719/5dcf34a.
        at org.jacoco.agent.rt.internal_b6258fc.CoverageTransformer.transform(CoverageTransformer.java:94)
        at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:244)
        at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
        at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:541)
        at java.base/java.lang.ClassLoader.defineClass2(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1104)
        at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:182)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:821)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassInModuleOrNull(BuiltinClassLoader.java:741)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClass(BuiltinClassLoader.java:621)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:633)
        at java.base/java.lang.Class.forName(Class.java:585)
        at java.base/java.lang.Class.forName(Class.java:560)
        at java.base/java.util.ServiceLoader.loadProvider(ServiceLoader.java:859)
        at java.base/java.util.ServiceLoader$ModuleServicesLookupIterator.hasNext(ServiceLoader.java:1084)
        at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)
        at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)
        at java.base/sun.util.cldr.CLDRLocaleProviderAdapter.lambda$new$0(CLDRLocaleProviderAdapter.java:84)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:571)
        at java.base/sun.util.cldr.CLDRLocaleProviderAdapter.<init>(CLDRLocaleProviderAdapter.java:83)
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484)
        at java.base/sun.util.locale.provider.LocaleProviderAdapter.forType(LocaleProviderAdapter.java:189)
        at java.base/sun.util.locale.provider.LocaleProviderAdapter.findAdapter(LocaleProviderAdapter.java:288)
        at java.base/sun.util.locale.provider.LocaleProviderAdapter.getAdapter(LocaleProviderAdapter.java:259)
        at java.base/java.util.Calendar.createCalendar(Calendar.java:1687)
        at java.base/java.util.Calendar.getInstance(Calendar.java:1655)
        at java.base/java.text.SimpleDateFormat.initializeCalendar(SimpleDateFormat.java:680)
        at java.base/java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:624)
        at java.base/java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:603)
        at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.getLastResortErrorLogFile(SystemApplicationClassLoaderWorker.java:129)
        at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:97)
        at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
        at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
        at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.io.IOException: Error while instrumenting sun/util/resources/cldr/provider/CLDRLocaleDataMetaInfo with JaCoCo 0.8.8.202204050719/5dcf34a.
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrumentError(Instrumenter.java:161)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:111)
        at org.jacoco.agent.rt.internal_b6258fc.CoverageTransformer.transform(CoverageTransformer.java:92)
        ... 35 more
Caused by: java.lang.IllegalArgumentException: Unsupported class file major version 64
        at org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader.<init>(ClassReader.java:199)
        at org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader.<init>(ClassReader.java:180)
        at org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader.<init>(ClassReader.java:166)
        at org.jacoco.agent.rt.internal_b6258fc.core.internal.instr.InstrSupport.classReaderFor(InstrSupport.java:280)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:77)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:109)
        ... 36 more

Gradle Test Executor 1 started executing tests.

> Task :opensearch-ml-algorithms:test

org.opensearch.ml.engine.MLEngineClassLoaderTests > initInstance_LocalSampleCalculator_RegisterFirst STANDARD_ERROR
    java.lang.instrument.IllegalClassFormatException: Error while instrumenting com/sun/tools/attach/VirtualMachine with JaCoCo 0.8.8.202204050719/5dcf34a.
        at org.jacoco.agent.rt.internal_b6258fc.CoverageTransformer.transform(CoverageTransformer.java:94)
        at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:244)
        at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
        at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:541)
        at java.base/java.lang.ClassLoader.defineClass2(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1104)
        at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:182)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:821)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassInModuleOrNull(BuiltinClassLoader.java:741)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:665)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:496)
        at java.base/java.lang.Class.forName(Class.java:475)
        at net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Accessor$Simple.of(ByteBuddyAgent.java:972)
        at net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$ForModularizedVm.attempt(ByteBuddyAgent.java:1096)
        at net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound.attempt(ByteBuddyAgent.java:1276)
        at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:626)
        at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:612)
        at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:564)
        at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:541)
        at org.mockito.internal.creation.bytebuddy.InlineDelegateByteBuddyMockMaker.<clinit>(InlineDelegateByteBuddyMockMaker.java:115)
        at org.mockito.internal.creation.bytebuddy.InlineByteBuddyMockMaker.<init>(InlineByteBuddyMockMaker.java:23)
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484)
        at org.mockito.internal.configuration.plugins.PluginInitializer.loadImpl(PluginInitializer.java:53)
        at org.mockito.internal.configuration.plugins.PluginLoader.loadPlugin(PluginLoader.java:69)
        at org.mockito.internal.configuration.plugins.PluginLoader.loadPlugin(PluginLoader.java:54)
        at org.mockito.internal.configuration.plugins.PluginRegistry.<init>(PluginRegistry.java:28)
        at org.mockito.internal.configuration.plugins.Plugins.<clinit>(Plugins.java:22)
        at org.mockito.internal.MockitoCore.<clinit>(MockitoCore.java:77)
        at org.mockito.Mockito.<clinit>(Mockito.java:1614)
        at org.opensearch.ml.engine.MLEngineClassLoaderTests.initInstance_LocalSampleCalculator_RegisterFirst(MLEngineClassLoaderTests.java:62)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
        at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
        at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
        at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
        at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
        at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
        at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60)
        at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
        at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
        at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
        at jdk.proxy2/jdk.proxy2.$Proxy5.processTestClass(Unknown Source)
        at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
        at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
        at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
        at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
        at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
        at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
        at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
        at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
        at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
    Caused by: java.io.IOException: Error while instrumenting com/sun/tools/attach/VirtualMachine with JaCoCo 0.8.8.202204050719/5dcf34a.
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrumentError(Instrumenter.java:161)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:111)
        at org.jacoco.agent.rt.internal_b6258fc.CoverageTransformer.transform(CoverageTransformer.java:92)
        ... 74 more
    Caused by: java.lang.IllegalArgumentException: Unsupported class file major version 64
        at org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader.<init>(ClassReader.java:199)
        at org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader.<init>(ClassReader.java:180)
        at org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader.<init>(ClassReader.java:166)
        at org.jacoco.agent.rt.internal_b6258fc.core.internal.instr.InstrSupport.classReaderFor(InstrSupport.java:280)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:77)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:109)
        ... 75 more
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

org.opensearch.ml.engine.MLEngineClassLoaderTests > initInstance_LocalSampleCalculator_RegisterFirst FAILED
    org.mockito.exceptions.base.MockitoException: 
    Mockito cannot mock this class: interface org.opensearch.client.Client.

    If you're not sure why you're getting this error, please report to the mailing list.

    Java               : 20
    JVM vendor name    : Amazon.com Inc.
    JVM vendor version : 20.0.1+9-FR
    JVM name           : OpenJDK 64-Bit Server VM
    JVM version        : 20.0.1+9-FR
    JVM info           : mixed mode, sharing
    OS name            : Mac OS X
    OS version         : 13.4

    You are seeing this disclaimer because Mockito is configured to create inlined mocks.
    You can learn about inline mocks and their limitations under item #39 of the Mockito class javadoc.

    Underlying exception : org.mockito.exceptions.base.MockitoException: Could not modify all classes [interface org.opensearch.client.OpenSearchClient, interface org.opensearch.client.Client, interface java.io.Closeable, interface java.lang.AutoCloseable, interface org.opensearch.core.common.lease.Releasable]
        at app//org.opensearch.ml.engine.MLEngineClassLoaderTests.initInstance_LocalSampleCalculator_RegisterFirst(MLEngineClassLoaderTests.java:62)

        Caused by:
        org.mockito.exceptions.base.MockitoException: Could not modify all classes [interface org.opensearch.client.OpenSearchClient, interface org.opensearch.client.Client, interface java.io.Closeable, interface java.lang.AutoCloseable, interface org.opensearch.core.common.lease.Releasable]
            at app//net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:157)
            at app//net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:371)
            at app//net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:179)
            at app//net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:382)
            ... 1 more

            Caused by:
            java.lang.IllegalStateException: 
            Byte Buddy could not instrument all classes within the mock's type hierarchy

            This problem should never occur for javac-compiled classes. This problem has been observed for classes that are:
             - Compiled by older versions of scalac
             - Classes that are part of the Android distribution
                at org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.triggerRetransformation(InlineBytecodeGenerator.java:280)
                at org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.mockClass(InlineBytecodeGenerator.java:213)
                at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator.lambda$mockClass$0(TypeCachingBytecodeGenerator.java:47)
                at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:157)
                at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:371)
                at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:179)
                at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:382)
                at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator.mockClass(TypeCachingBytecodeGenerator.java:40)
                at org.mockito.internal.creation.bytebuddy.InlineDelegateByteBuddyMockMaker.createMockType(InlineDelegateByteBuddyMockMaker.java:389)
                at org.mockito.internal.creation.bytebuddy.InlineDelegateByteBuddyMockMaker.doCreateMock(InlineDelegateByteBuddyMockMaker.java:349)
                at org.mockito.internal.creation.bytebuddy.InlineDelegateByteBuddyMockMaker.createMock(InlineDelegateByteBuddyMockMaker.java:328)
                at org.mockito.internal.creation.bytebuddy.InlineByteBuddyMockMaker.createMock(InlineByteBuddyMockMaker.java:56)
                at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:53)
                at org.mockito.internal.MockitoCore.mock(MockitoCore.java:96)
                at org.mockito.Mockito.mock(Mockito.java:1965)
                at org.mockito.Mockito.mock(Mockito.java:1880)
                ... 1 more

                Caused by:
                java.lang.IllegalArgumentException: Java 20 (64) is not supported by the current version of Byte Buddy which officially supports Java 18 (62) - update Byte Buddy or set net.bytebuddy.experimental as a VM property
                    at net.bytebuddy.utility.OpenedClassReader.of(OpenedClassReader.java:96)
                    at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining.create(TypeWriter.java:3948)
                    at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2213)
                    at net.bytebuddy.dynamic.scaffold.inline.RedefinitionDynamicTypeBuilder.make(RedefinitionDynamicTypeBuilder.java:224)
                    at net.bytebuddy.dynamic.scaffold.inline.AbstractInliningDynamicTypeBuilder.make(AbstractInliningDynamicTypeBuilder.java:123)
                    at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase.make(DynamicType.java:3661)
                    at org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.transform(InlineBytecodeGenerator.java:394)
                    at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:244)
                    at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
                    at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:541)
                    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
                    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:169)
                    at org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.triggerRetransformation(InlineBytecodeGenerator.java:276)
                    ... 16 more

Mainly the issue i think:

Caused by:
                java.lang.IllegalArgumentException: Java 20 (64) is not supported by the current version of Byte Buddy which officially supports Java 18 (62) - update Byte Buddy or set net.bytebuddy.experimental as a VM property
b4sjoo commented 1 year ago

Yeah it seems there will be many breaking changes after migrating to jdk20, sigh...

dhrubo-os commented 1 year ago

Update:

We can reproduce the same error in our end with executing: ./gradlew :opensearch-ml-plugin:delombok


The :opensearch-ml-plugin:delombok task failed.View task in console log
Process 'command '/Library/Java/JavaVirtualMachines/amazon-corretto-20.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1
Exception
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':opensearch-ml-plugin:delombok'.    
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:149)  
    at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282) 
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:147)   
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:135)  
    at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)   
    at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)   
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57) 
    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:74)   
    at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)   
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)   
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)  
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)  
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157) 
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)  
    at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53) 
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73) 
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52) 
    at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)   
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:337)   
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:324)   
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:317)  
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)  
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463)   
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380)   
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)   
    at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)    
Caused by: org.gradle.process.internal.ExecException: Process 'command '/Library/Java/JavaVirtualMachines/amazon-corretto-20.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1   
    at org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNormalExitValue(DefaultExecHandle.java:415)   
    at org.gradle.process.internal.DefaultJavaExecAction.execute(DefaultJavaExecAction.java:52) 
    at org.gradle.process.internal.DefaultExecActionFactory.javaexec(DefaultExecActionFactory.java:195) 
    at org.gradle.process.internal.DefaultExecOperations.javaexec(DefaultExecOperations.java:42)    
    at io.freefair.gradle.plugins.lombok.tasks.Delombok.delombok(Delombok.java:198) 
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)  
    at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)   
    at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58) 
    at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)   
    at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)   
    at org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:242)  
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157) 
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)  
    at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)  
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)  
    at org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:227)  
    at org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:210) 
    at org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:193) 
    at org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:166)    
    at org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:105)    
    at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:44)  
    at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:59)  
    at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:56)  
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157) 
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)  
    at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53) 
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73) 
    at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:56) 
    at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:44) 
    at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:67) 
    at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:37) 
    at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41) 
    at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)   
    at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55) 
    at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:50) 
    at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:28) 
    at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.executeDelegateBroadcastingChanges(CaptureStateAfterExecutionStep.java:100)   
    at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:72)   
    at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:50)   
    at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:40) 
    at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:29) 
    at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:166)  
    at org.gradle.internal.execution.steps.BuildCacheStep.lambda$execute$1(BuildCacheStep.java:70)  
    at org.gradle.internal.Either$Right.fold(Either.java:175)   
    at org.gradle.internal.execution.caching.CachingState.fold(CachingState.java:59)    
    at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:68)   
    at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:46)   
    at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:36) 
    at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:25) 
    at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:36) 
    at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:22) 
    at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:91)    
    at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$2(SkipUpToDateStep.java:55)  
    at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:55)   
    at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:37)   
    at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:65)   
    at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:36)   
    at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)    
    at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)    
    at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:76) 
    at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:37) 
    at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:94)   
    at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:49)   
    at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:71) 
    at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:45) 
    at org.gradle.internal.execution.steps.SkipEmptyWorkStep.executeWithNonEmptySources(SkipEmptyWorkStep.java:177) 
    at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:86) 
    at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:53) 
    at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:32) 
    at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:21) 
    at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)  
    at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:36)   
    at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:23)   
    at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:75) 
    at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:41) 
    at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:32)    
    at org.gradle.api.internal.tasks.execution.TaskExecution$4.withWorkspace(TaskExecution.java:287)    
    at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:30) 
    at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:21) 
    at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:37) 
    at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:27) 
    at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:47)   
    at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:34)   
    at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:64)  
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:146)   
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:135)  
    at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)   
    at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)   
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57) 
    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:74)   
    at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)   
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)   
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)  
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)  
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)    
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157) 
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)  
    at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53) 
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73) 
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52) 
    at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)   
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:337)   
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:324)   
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:317)  
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)  
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463)   
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380)   
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)   
    at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
reta commented 1 year ago

Yeah it seems there will be many breaking changes after migrating to jdk20, sigh...

Do you folks need help with that? The lombok is real bummer for any recent JDK updates, Mockito 5.x has no issues with JDK-20

dhrubo-os commented 1 year ago

@reta Right now OS is building failing due to delombok issue, we created an issue in freefair.

If you have any suggestion about the delombok issue that will be helpful.

In addition, we are facing build error:

Caused by: java.io.IOException: Error while instrumenting sun/util/resources/cldr/provider/CLDRLocaleDataMetaInfo with JaCoCo 0.8.8.202204050719/5dcf34a.
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrumentError(Instrumenter.java:161)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:111)
        at org.jacoco.agent.rt.internal_b6258fc.CoverageTransformer.transform(CoverageTransformer.java:92)
        ... 35 more
Caused by: java.lang.IllegalArgumentException: Unsupported class file major version 64

We are adding jacoco dependencies here.

If you have any suggestion how to solve this issue, that will also be helpful.

Thanks.

reta commented 1 year ago

@dhrubo-os anything left here you need help with folks? or we could close it :)

dhrubo-os commented 1 year ago

@reta I think that's all the support we needed for now. Thanks a lot. I'm waiting for a successful build for OS build issue. After confirming this, I'll close the issue.

dhrubo-os commented 1 year ago

Tested the build in my end and the build was successful. Closing the issue. Thanks.

b4sjoo commented 1 year ago

Here is a temporary fix method: opensearch-project/security#2782

1191 Provide a final solution to string methods

b4sjoo commented 1 year ago

More refactoring from OS core - https://github.com/opensearch-project/OpenSearch/pull/8157 Fixed in PR #1122