Add Windows CI on pull requests

dblock commented 2 years ago

Is your feature request related to a problem? Please describe. Coming from https://github.com/opensearch-project/opensearch-build/issues/33

[x] Precommit
[x] https://github.com/opensearch-project/OpenSearch/issues/4811 CI support Windows
[ ] Run a subset of tests on PRs on Windows

Gradle Check running on Windows (Failures so far)

[x] Permissions failures
[x] Java Home setup for: 11,8, 9, 10,,12,13,14,17
[x] Cross compilation errors for *nix distributions
[x] Build Tools integTests
[x] Gradle Reaper process failing
[x] #5324
[x] #5323
[x] #5400
[x] https://github.com/opensearch-project/opensearch-build/issues/3052
[x] https://github.com/opensearch-project/opensearch-build-libraries/issues/113
[ ] https://github.com/opensearch-project/opensearch-build-libraries/issues/114

Describe the solution you'd like Add Windows CI for this repo at least for the minimum gradle precommit.

dblock commented 1 year ago

How much of the tests is precommit running? For example, will tests in https://github.com/opensearch-project/OpenSearch/pull/4656 be exercised in CI with the change, or did we close this prematurely?

dblock commented 1 year ago

@saratvemulapalli @dbwiddis ^

saratvemulapalli commented 1 year ago

How much of the tests is precommit running? For example, will tests in #4656 be exercised in CI with the change, or did we close this prematurely?

@dblock thats a good point. Precommit only runs few basic checks but does not run unit/integ tests. I'll re-open this to add gradle check support for Windows.

Thinking about next steps, I am worried if we do have multiple gradle checks (linux, windows) the chances of failure multiplies (unfortunately due to flaky tests).

How about splitting platform specific tests and only run them on windows/macos etc instead of running the whole suite ? Also do you know how other projects take care of this?

dblock commented 1 year ago

AFAIK the majority of flakey tests are integration and it's a separate (albeit important) problem. I believe there's a set of unit tests that never fails and we should start with that and keep increasing coverage. Starting with platform-specific tests is a good idea, I would gladly merge just that!

saratvemulapalli commented 1 year ago

I've spent sometime yesterday was able to run gradle check on windows, tests are failing due to permissions errors.

org.opensearch.index.mapper.ICUCollationKeywordFieldMapperTests > classMethod FAILED
    java.lang.RuntimeException: can not run opensearch as root
        at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:124)
        at org.opensearch.bootstrap.BootstrapForTesting.<clinit>(BootstrapForTesting.java:105)
        at org.opensearch.test.OpenSearchTestCase.<clinit>(OpenSearchTestCase.java:257)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:467)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:623)

Suite: Test class org.opensearch.index.mapper.ICUCollationKeywordFieldMapperTests
  2> java.lang.RuntimeException: can not run opensearch as root
        at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:124)
        at org.opensearch.bootstrap.BootstrapForTesting.<clinit>(BootstrapForTesting.java:105)
        at org.opensearch.test.OpenSearchTestCase.<clinit>(OpenSearchTestCase.java:257)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:467)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:623)
Exception in thread "Thread-4" java.lang.NoClassDefFoundError: Could not initialize class org.opensearch.test.OpenSearchTestCase
        at java.base/java.lang.Thread.run(Thread.java:833)
        Suppressed: java.lang.IllegalStateException: No context information for thread: Thread[id=33, name=Thread-4, state=RUNNABLE, group=TGRP-ICUCollationKeywordFieldMapperTests]. Is this thread running under a class com.carrotsearch.randomizedtesting.RandomizedRunner runner context? Add @RunWith(class com.carrotsearch.randomizedtesting.RandomizedRunner.class) to your test class. Make sure your code accesses random contexts within @BeforeClass and @AfterClass boundary (for example, static test class initializers are not permitted to access random contexts).
                at com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:249)
                at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
                at com.carrotsearch.randomizedtesting.RandomizedRunner.augmentStackTrace(RandomizedRunner.java:1885)
                at com.carrotsearch.randomizedtesting.RunnerThreadGroup.uncaughtException(RunnerThreadGroup.java:20)
                at java.base/java.lang.Thread.dispatchUncaughtException(Thread.java:2008)

Tests with failures:
 - org.opensearch.index.analysis.SimpleIcuAnalysisTests.classMethod
 - org.opensearch.index.analysis.SimpleIcuNormalizerCharFilterTests.classMethod
 - org.opensearch.index.mapper.ICUCollationKeywordFieldMapperTests.classMethod
 - org.opensearch.index.mapper.ICUCollationKeywordFieldTypeTests.classMethod
 - org.opensearch.index.analysis.AnalysisICUFactoryTests.classMethod
 - org.opensearch.index.analysis.IcuTokenizerFactoryTests.classMethod
 - org.opensearch.index.analysis.SimpleIcuCollationTokenFilterTests.classMethod
 - org.opensearch.index.mapper.CollationFieldTypeTests.classMethod

saratvemulapalli commented 1 year ago

I was able to get past the problem of permissions. Looks like we have to run as a non-root user. Now its bunch of new failures related to BWC and checking out old code.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':distribution:bwc:minor:checkoutBwcBranch'.
> Process 'command 'git'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':distribution:bwc:bugfix:checkoutBwcBranch'.
> Process 'command 'git'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 3m 54s
1275 actionable tasks: 942 executed, 333 up-to-date
PS C:\Users\Sarat\Desktop\Workspace\OpenSearch>

saratvemulapalli commented 1 year ago

Got past the problems with bwc now hitting problems with Java mmap.

----- Last  20 lines from daemon log file - daemon-29408.out.log -----
# Native memory allocation (mmap) failed to map 65536 bytes for Failed to commit metaspace.
# An error report file with more information is saved as:
# C:\Users\Sarat\Desktop\Workspace\OpenSearch\modules\analysis-common\build\testrun\test\hs_err_pid13212.log
# An error report file with more information is saved as:
# C:\Users\Sarat\Desktop\Workspace\OpenSearch\modules\ingest-common\build\testrun\test\hs_err_pid24072.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for Failed to commit metaspace.
# An error report file with more information is saved as:
# C:\Users\Sarat\Desktop\Workspace\OpenSearch\modules\analysis-common\build\testrun\test\hs_err_pid4164.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for Failed to commit metaspace.
# An error report file with more information is saved as:
# C:\Users\Sarat\Desktop\Workspace\OpenSearch\libs\core\build\testrun\test\hs_err_pid11592.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for Failed to commit metaspace.
# An error report file with more information is saved as:
# C:\Users\Sarat\Desktop\Workspace\OpenSearch\modules\reindex\build\testrun\test\hs_err_pid23916.log

peterzhuamazon commented 1 year ago

Hi,

@dblock @saratvemulapalli could you guys let me know what is the reason that Gradle Check is needed for Windows? As of current state, the existing gradle check is already running extensively long, and it only able to run on Ubuntu server. There are also many other utilities such as docker/compose needs to be setup before we even do any testings on it.

I believe the windows ci should be the building process on Windows, which we can then use github actions to achieve, not gradle check.

Please feel free to let me know but this requires a lot of effort just to setup the runner. And the existing windows runner is not suitable for this purpose, as it is sorely built to run builds, not gradle check.

Thanks.

peterzhuamazon commented 1 year ago

We need to plan this from our side if we bound to add windows gradle check. CPU limitations, dynamic runner switch, AMI creation, Docker/Compose on Windows, and many more.

peterzhuamazon commented 1 year ago

Also, Gradle check already takes a long time, and every windows agent takes around 10-20min just to bootstrap on Jenkins. Can we make sure gradle check heavyweight tasks are de-coupled before we add another gradle check?

saratvemulapalli commented 1 year ago

@peterzhuamazon we cannot release software which we dont test. We have to make sure OpenSearch on windows is being tested to release the software, the way we do is via gradle check.

peterzhuamazon commented 1 year ago

@peterzhuamazon we cannot release software which we dont test. We have to make sure OpenSearch on windows is being tested to release the software, the way we do is via gradle check.

Got it. Lets talk a bit more on the requirements. The existing Jenkins workflow can take a new param to define whether it is linux or windows, and switch runner based on the trigger.

The thing I am more concerning is the windows runner takes 2x time to complete gradle check compares to linux host.

peterzhuamazon commented 1 year ago

Just talked with @saratvemulapalli and @Poojita-Raj and here are the steps on high level:

Edit github workflow to have a new workflow file for Windows trigger, on cron, on manual. It should have a new param to enter the platform of the test.
Edit Jenkinsfile to take the platform param and able to switch runner on the fly.
Add new packer profile to create a new Windows AMI specifically for gradle check, adding docker, compose, etc.
Adding new windows agent profile based on the AMI in cdk.
Deploy the new cdk into the public jenkins.
Pull the 1 2 into Jenkins and try to test by manually triggering the workflow and fix until it passes.
Adjust parallel runs number and apply for more hardware for the task.

vibrantvarun commented 1 year ago

Hi, The steps to setup OpenSearch project before running gradlew check. 1) Do not admin user cmd (Command Line prompt). Use any non-root user cmd. 2) Set java 11,8, 9, 10,,12,13,14,17 paths as JAVA_HOME, JAVA8_HOME..etc respectively in system variables and also configure all the above paths in paths environment variable 3) clone the project 4) gradlew clean 5) gradlew check

vibrantvarun commented 1 year ago

Facing the below errors.

`1: Task failed with an exception.

What went wrong: Execution failed for task ':distribution:archives:darwin-arm64-tar:checkExtraction'.

Could not read 'C:\Users\Varun\OpenSearch\distribution\archives\darwin-arm64-tar\build\distributions\opensearch-min-2.3.1-SNAPSHOT-darwin-arm64.tar.gz' as it does not exist.
Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

2: Task failed with an exception.

What went wrong: Execution failed for task ':distribution:archives:freebsd-tar:checkExtraction'.

Could not read 'C:\Users\Varun\OpenSearch\distribution\archives\freebsd-tar\build\distributions\opensearch-min-2.3.1-SNAPSHOT-freebsd-x64.tar.gz' as it does not exist.
Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

3: Task failed with an exception.

What went wrong: Execution failed for task ':distribution:archives:darwin-tar:checkExtraction'.

Could not read 'C:\Users\Varun\OpenSearch\distribution\archives\darwin-tar\build\distributions\opensearch-min-2.3.1-SNAPSHOT-darwin-x64.tar.gz' as it does not exist.
Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

4: Task failed with an exception.

What went wrong: org.gradle.api.GradleException: Reaper process failed. Check log at C:\Users\Varun\OpenSearch.gradle\reaper\build-14320\error.log for details

Reaper process failed. Check log at C:\Users\Varun\OpenSearch.gradle\reaper\build-14320\error.log for details
Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
Get more help at https://help.gradle.org

BUILD FAILED in 17m 40s 2327 actionable tasks: 2272 executed, 40 from cache, 15 up-to-date`

saratvemulapalli commented 1 year ago

Build Tools, integTest is failing. Tests look very platform specific.


> Task :build-tools:integTest

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testTar FAILED
    java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testBZip2Tar FAILED
    java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testBZip2TarDoNotPreserveFileTimestamps FAILED
    java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testTarDoNotPreserveFileTimestamps FAILED
    java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testGZipTar FAILED
    java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testGZipTarDoNotPreserveFileTimestamps FAILED
    java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

saratvemulapalli commented 1 year ago

Finally I was able to run gradle check with 2+ hrs of runtime

Tests with failures:
 - org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation
 - org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.classMethod

11295 tests completed, 12 failed, 31 skipped

> Task :server:test FAILED

FAILURE: Build completed with 7 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':distribution:archives:darwin-arm64-tar:checkExtraction'.
> Could not read 'C:\Users\Sarat\Desktop\Workspace\OpenSearch\distribution\archives\darwin-arm64-tar\build\distributions\opensearch-min-3.0.0-SNAPSHOT-darwin-arm64.tar.gz' as it does not exist.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':distribution:archives:darwin-tar:checkExtraction'.
> Could not read 'C:\Users\Sarat\Desktop\Workspace\OpenSearch\distribution\archives\darwin-tar\build\distributions\opensearch-min-3.0.0-SNAPSHOT-darwin-x64.tar.gz' as it does not exist.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================

3: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':distribution:archives:freebsd-tar:checkExtraction'.
> Could not read 'C:\Users\Sarat\Desktop\Workspace\OpenSearch\distribution\archives\freebsd-tar\build\distributions\opensearch-min-3.0.0-SNAPSHOT-freebsd-x64.tar.gz' as it does not exist.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================

4: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':distribution:archives:linux-arm64-tar:checkExtraction'.
> Could not read 'C:\Users\Sarat\Desktop\Workspace\OpenSearch\distribution\archives\linux-arm64-tar\build\distributions\opensearch-min-3.0.0-SNAPSHOT-linux-arm64.tar.gz' as it does not exist.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================

5: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':build-tools:integTest'.
> There were failing tests. See the report at: file:///C:/Users/Sarat/Desktop/Workspace/OpenSearch/buildSrc/build/reports/tests/integTest/index.html

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================

6: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':server:test'.
> There were failing tests. See the report at: file:///C:/Users/Sarat/Desktop/Workspace/OpenSearch/server/build/reports/tests/test/index.html

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================

7: Task failed with an exception.
-----------
* What went wrong:
org.gradle.api.GradleException: Reaper process failed. Check log at C:\Users\Sarat\Desktop\Workspace\OpenSearch\.gradle\reaper\build-4316\error.log for details
> Reaper process failed. Check log at C:\Users\Sarat\Desktop\Workspace\OpenSearch\.gradle\reaper\build-4316\error.log for details

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 3m 34s
2343 actionable tasks: 393 executed, 1950 up-to-date

vibrantvarun commented 1 year ago

Below failures comes after 96%+ build of gradle check. I have made some modifications regarding commenting code of every possible test case running on any other OS except windows.

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
------------cluster:v2.4.0#mixedClusterTest > Executing test org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT
* What went wrong:
Execution failed for task ':modules:reindex:yamlRestTest'.
> There were failing tests. See the report at: file:///C:/Users/Varun/OpenSearch/modules/reindex/build/reports/tests/yamlRestTest/index.html

* Try:
> Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':modules:reindex:yamlRestTest'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:142)
        at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:140)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:128)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:69)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:327)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:314)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:307)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:293)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:420)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:342)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
Caused by: org.gradle.api.tasks.VerificationException: There were failing tests. See the report at: file:///C:/Users/Varun/OpenSearch/modules/reindex/build/reports/tests/yamlRestTest/index.html
        at org.gradle.api.tasks.testing.AbstractTestTask.handleTestFailures(AbstractTestTask.java:651)
        at org.gradle.api.tasks.testing.AbstractTestTask.handleCollectedResults(AbstractTestTask.java:513)
        at org.gradle.api.tasks.testing.AbstractTestTask.executeTests(AbstractTestTask.java:508)
        at org.gradle.api.tasks.testing.Test.executeTests(Test.java:690)
        at jdk.internal.reflect.GeneratedMethodAccessor2110.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
        at org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:236)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
        at org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:221)
        at org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:204)
        at org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:187)
        at org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:165)
        at org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:89)
        at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:40)
        at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:53)
        at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:50)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
        at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:50)
        at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:40)
        at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:68)
        at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:38)
        at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41)
        at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)
        at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55)
        at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:51)
        at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:29)
        at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.executeDelegateBroadcastingChanges(CaptureStateAfterExecutionStep.java:124)
        at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:80)
        at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:58)
        at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:48)
        at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:36)
        at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:181)
        at org.gradle.internal.execution.steps.BuildCacheStep.executeAndStoreInCache(BuildCacheStep.java:154)
        at org.gradle.internal.execution.steps.BuildCacheStep.lambda$executeWithCache$4(BuildCacheStep.java:121)
        at org.gradle.internal.execution.steps.BuildCacheStep.lambda$executeWithCache$5(BuildCacheStep.java:121)
        at org.gradle.internal.Try$Success.map(Try.java:164)
        at org.gradle.internal.execution.steps.BuildCacheStep.executeWithCache(BuildCacheStep.java:81)
        at org.gradle.internal.execution.steps.BuildCacheStep.lambda$execute$0(BuildCacheStep.java:70)
        at org.gradle.internal.Either$Left.fold(Either.java:115)
        at org.gradle.internal.execution.caching.CachingState.fold(CachingState.java:59)
        at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:69)
        at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:47)
        at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:36)
        at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:25)
        at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:36)
        at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:22)
        at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:110)
        at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$2(SkipUpToDateStep.java:56)
        at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:56)
        at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:38)
        at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:73)
        at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:44)
        at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)
        at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)
        at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:89)
        at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:50)
        at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:114)
        at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:57)
        at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:76)
        at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:50)
        at org.gradle.internal.execution.steps.SkipEmptyWorkStep.executeWithNoEmptySources(SkipEmptyWorkStep.java:254)
        at org.gradle.internal.execution.steps.SkipEmptyWorkStep.executeWithNoEmptySources(SkipEmptyWorkStep.java:209)
        at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:88)
        at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:56)
        at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:32)
        at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:21)
        at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)
        at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:43)
        at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:31)
        at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:40)
        at org.gradle.api.internal.tasks.execution.TaskExecution$4.withWorkspace(TaskExecution.java:281)
        at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:40)
        at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:30)
        at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:37)
        at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:27)
        at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:44)
        at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:33)
        at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:76)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:139)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:128)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:69)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:327)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:314)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:307)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:293)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:420)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:342)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)

==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
org.gradle.api.GradleException: Reaper process failed. Check log at C:\Users\Varun\OpenSearch\.gradle\reaper\build-7972\error.log for details
> Reaper process failed. Check log at C:\Users\Varun\OpenSearch\.gradle\reaper\build-7972\error.log for details

* Try:
> Run with --info or --debug option to get more log output.

* Exception is:
java.lang.RuntimeException: org.gradle.api.GradleException: Reaper process failed. Check log at C:\Users\Varun\OpenSearch\.gradle\reaper\build-7972\error.log for details
        at org.opensearch.gradle.ReaperService.shutdown(ReaperService.java:118)
        at org.opensearch.gradle.ReaperPlugin.lambda$apply$0(ReaperPlugin.java:63)
        at org.gradle.internal.event.BroadcastDispatch$ActionInvocationHandler.dispatch(BroadcastDispatch.java:95)
        at org.gradle.internal.event.BroadcastDispatch$ActionInvocationHandler.dispatch(BroadcastDispatch.java:83)
        at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:43)
        at org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:245)
        at org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:157)
        at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:61)
        at org.gradle.internal.event.BroadcastDispatch$CompositeDispatch.dispatch(BroadcastDispatch.java:346)
        at org.gradle.internal.event.BroadcastDispatch$CompositeDispatch.dispatch(BroadcastDispatch.java:249)
        at org.gradle.internal.event.ListenerBroadcast.dispatch(ListenerBroadcast.java:141)
        at org.gradle.internal.event.ListenerBroadcast.dispatch(ListenerBroadcast.java:37)
        at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
        at com.sun.proxy.$Proxy28.buildFinished(Unknown Source)
        at org.gradle.internal.build.DefaultBuildLifecycleController.lambda$finishBuild$8(DefaultBuildLifecycleController.java:191)
        at org.gradle.internal.model.StateTransitionController.lambda$finish$11(StateTransitionController.java:227)
        at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:44)
        at org.gradle.internal.model.StateTransitionController.finish(StateTransitionController.java:218)
        at org.gradle.internal.build.DefaultBuildLifecycleController.finishBuild(DefaultBuildLifecycleController.java:179)
        at org.gradle.internal.buildtree.DefaultBuildTreeFinishExecutor.finishBuildTree(DefaultBuildTreeFinishExecutor.java:55)
        at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.lambda$runBuild$4(DefaultBuildTreeLifecycleController.java:111)
        at org.gradle.internal.model.StateTransitionController.lambda$transition$5(StateTransitionController.java:166)
        at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:247)
        at org.gradle.internal.model.StateTransitionController.lambda$transition$6(StateTransitionController.java:166)
        at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:44)
        at org.gradle.internal.model.StateTransitionController.transition(StateTransitionController.java:166)
        at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.runBuild(DefaultBuildTreeLifecycleController.java:103)
        at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.scheduleAndRunTasks(DefaultBuildTreeLifecycleController.java:69)
        at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:31)
        at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
        at org.gradle.internal.buildtree.ProblemReportingBuildActionRunner.run(ProblemReportingBuildActionRunner.java:49)
        at org.gradle.launcher.exec.BuildOutcomeReportingBuildActionRunner.run(BuildOutcomeReportingBuildActionRunner.java:65)
        at org.gradle.tooling.internal.provider.FileSystemWatchingBuildActionRunner.run(FileSystemWatchingBuildActionRunner.java:136)
        at org.gradle.launcher.exec.BuildCompletionNotifyingBuildActionRunner.run(BuildCompletionNotifyingBuildActionRunner.java:41)
        at org.gradle.launcher.exec.RootBuildLifecycleBuildActionExecutor.lambda$execute$0(RootBuildLifecycleBuildActionExecutor.java:40)
        at org.gradle.composite.internal.DefaultRootBuildState.run(DefaultRootBuildState.java:122)
        at org.gradle.launcher.exec.RootBuildLifecycleBuildActionExecutor.execute(RootBuildLifecycleBuildActionExecutor.java:40)
        at org.gradle.internal.buildtree.DefaultBuildTreeContext.execute(DefaultBuildTreeContext.java:40)
        at org.gradle.launcher.exec.BuildTreeLifecycleBuildActionExecutor.lambda$execute$0(BuildTreeLifecycleBuildActionExecutor.java:65)
        at org.gradle.internal.buildtree.BuildTreeState.run(BuildTreeState.java:53)
        at org.gradle.launcher.exec.BuildTreeLifecycleBuildActionExecutor.execute(BuildTreeLifecycleBuildActionExecutor.java:65)
        at org.gradle.launcher.exec.RunAsBuildOperationBuildActionExecutor$3.call(RunAsBuildOperationBuildActionExecutor.java:61)
        at org.gradle.launcher.exec.RunAsBuildOperationBuildActionExecutor$3.call(RunAsBuildOperationBuildActionExecutor.java:57)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
        at org.gradle.launcher.exec.RunAsBuildOperationBuildActionExecutor.execute(RunAsBuildOperationBuildActionExecutor.java:57)
        at org.gradle.launcher.exec.RunAsWorkerThreadBuildActionExecutor.lambda$execute$0(RunAsWorkerThreadBuildActionExecutor.java:36)
        at org.gradle.internal.work.DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:249)
        at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:109)
        at org.gradle.launcher.exec.RunAsWorkerThreadBuildActionExecutor.execute(RunAsWorkerThreadBuildActionExecutor.java:36)
        at org.gradle.tooling.internal.provider.continuous.ContinuousBuildActionExecutor.execute(ContinuousBuildActionExecutor.java:110)
        at org.gradle.tooling.internal.provider.SubscribableBuildActionExecutor.execute(SubscribableBuildActionExecutor.java:64)
        at org.gradle.internal.session.DefaultBuildSessionContext.execute(DefaultBuildSessionContext.java:46)
        at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter$ActionImpl.apply(BuildSessionLifecycleBuildActionExecuter.java:100)
        at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter$ActionImpl.apply(BuildSessionLifecycleBuildActionExecuter.java:88)
        at org.gradle.internal.session.BuildSessionState.run(BuildSessionState.java:69)
        at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter.execute(BuildSessionLifecycleBuildActionExecuter.java:62)
        at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter.execute(BuildSessionLifecycleBuildActionExecuter.java:41)
        at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:63)
        at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:31)
        at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:52)
        at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:40)
        at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:47)
        at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:31)
        at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:65)
        at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
        at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:39)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
        at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:29)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
        at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:35)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
        at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.create(ForwardClientInput.java:78)
        at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.create(ForwardClientInput.java:75)
        at org.gradle.util.internal.Swapper.swap(Swapper.java:38)
        at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:75)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
        at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:55)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
        at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:63)
        at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
        at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:84)
        at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
        at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:52)
        at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:297)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.gradle.api.GradleException: Reaper process failed. Check log at C:\Users\Varun\OpenSearch\.gradle\reaper\build-7972\error.log for details
        at org.opensearch.gradle.ReaperService.shutdown(ReaperService.java:115)
        ... 98 more

==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 57m 1s
2412 actionable tasks: 1515 executed, 559 from cache, 338 up-to-date
�

dblock commented 1 year ago

For this specific issue, we want some windows CI on GHA that runs on every PR. The complete gradle check with all integration tests is another ask.

vibrantvarun commented 1 year ago

TEST Failing while othe platforms code commented. 1)

MultiVersionRepositoryAccessIT. classMethod
MultiVersionRepositoryAccessIT. classMethod
MultiVersionRepositoryAccessIT. classMethod
MultiVersionRepositoryAccessIT. classMethod
MultiVersionRepositoryAccessIT. testCreateAndRestoreSnapshot
MultiVersionRepositoryAccessIT. testCreateAndRestoreSnapshot
MultiVersionRepositoryAccessIT. testCreateAndRestoreSnapshot
MultiVersionRepositoryAccessIT. testCreateAndRestoreSnapshot�

Test failure errors: a) java.lang.Exception: Suite timeout exceeded (>= 1800000 msec). at randomizedtesting.SeedInfo.seed([6614DC617C2EB13B]:0)� b) java.lang.Exception: Test abandoned because suite timeout was reached. at randomizedtesting.SeedInfo.seed([6614DC617C2EB13B]:0)�

2)

RepositoryURLClientYamlTestSuiteIT. classMethod
RepositoryURLClientYamlTestSuiteIT. classMethod
RepositoryURLClientYamlTestSuiteIT. classMethod
RepositoryURLClientYamlTestSuiteIT. classMethod
RepositoryURLClientYamlTestSuiteIT. test {yaml=repository_url/10_basic/Module repository-url is loaded}
RepositoryURLClientYamlTestSuiteIT. test {yaml=repository_url/10_basic/Module repository-url is loaded}
RepositoryURLClientYamlTestSuiteIT. test {yaml=repository_url/10_basic/Module repository-url is loaded}
RepositoryURLClientYamlTestSuiteIT. test {yaml=repository_url/10_basic/Module repository-url is loaded}�

Test failure errors: a) java.lang.Exception: Suite timeout exceeded (>= 1800000 msec). at randomizedtesting.SeedInfo.seed([6614DC617C2EB13B]:0)� b) java.lang.Exception: Test abandoned because suite timeout was reached. at randomizedtesting.SeedInfo.seed([6614DC617C2EB13B]:0)�

3)

PercolatorClientYamlTestSuiteIT. classMethod
PercolatorClientYamlTestSuiteIT. classMethod
PercolatorClientYamlTestSuiteIT. classMethod
PercolatorClientYamlTestSuiteIT. classMethod
PercolatorClientYamlTestSuiteIT. test {yaml=/10_basic/Test percolator basics via rest}
PercolatorClientYamlTestSuiteIT. test {yaml=/10_basic/Test percolator basics via rest}
PercolatorClientYamlTestSuiteIT. test {yaml=/10_basic/Test percolator basics via rest}
PercolatorClientYamlTestSuiteIT. test {yaml=/10_basic/Test percolator basics via rest}�

same errors and failure as above test

4)

ParentChildClientYamlTestSuiteIT. classMethod
ParentChildClientYamlTestSuiteIT. classMethod
ParentChildClientYamlTestSuiteIT. classMethod
ParentChildClientYamlTestSuiteIT. classMethod
ParentChildClientYamlTestSuiteIT. test {yaml=/40_aggregations/parent}
ParentChildClientYamlTestSuiteIT. test {yaml=/40_aggregations/parent}
ParentChildClientYamlTestSuiteIT. test {yaml=/40_aggregations/parent}
ParentChildClientYamlTestSuiteIT. test {yaml=/40_aggregations/parent}�

same errors and failure as above test

5)

SearchTemplateWithoutContentIT. classMethod
SearchTemplateWithoutContentIT. classMethod
SearchTemplateWithoutContentIT. classMethod
SearchTemplateWithoutContentIT. classMethod
SearchTemplateWithoutContentIT. testMultiSearchTemplateMissingBody
SearchTemplateWithoutContentIT. testMultiSearchTemplateMissingBody
SearchTemplateWithoutContentIT. testMultiSearchTemplateMissingBody
SearchTemplateWithoutContentIT. testMultiSearchTemplateMissingBody�

same errors and failure as above test

6)

IngestUserAgentClientYamlTestSuiteIT. classMethod
IngestUserAgentClientYamlTestSuiteIT. classMethod
IngestUserAgentClientYamlTestSuiteIT. classMethod
IngestUserAgentClientYamlTestSuiteIT. classMethod
IngestUserAgentClientYamlTestSuiteIT. test {yaml=ingest-useragent/30_custom_regex/Test user agent processor with custom regex file}
IngestUserAgentClientYamlTestSuiteIT. test {yaml=ingest-useragent/30_custom_regex/Test user agent processor with custom regex file}
IngestUserAgentClientYamlTestSuiteIT. test {yaml=ingest-useragent/30_custom_regex/Test user agent processor with custom regex file}
IngestUserAgentClientYamlTestSuiteIT. test {yaml=ingest-useragent/30_custom_regex/Test user agent processor with custom regex file}�

same errors and failure as above test

7)

ExampleRescoreClientYamlTestSuiteIT. classMethod
ExampleRescoreClientYamlTestSuiteIT. classMethod
ExampleRescoreClientYamlTestSuiteIT. classMethod
ExampleRescoreClientYamlTestSuiteIT. classMethod
ExampleRescoreClientYamlTestSuiteIT. test {yaml=example-rescore/10_basic/Plugin loaded}
ExampleRescoreClientYamlTestSuiteIT. test {yaml=example-rescore/10_basic/Plugin loaded}
ExampleRescoreClientYamlTestSuiteIT. test {yaml=example-rescore/10_basic/Plugin loaded}
ExampleRescoreClientYamlTestSuiteIT. test {yaml=example-rescore/10_basic/Plugin loaded}�

same errors and failures as above test

8)

ExpertScriptClientYamlTestSuiteIT. classMethod
ExpertScriptClientYamlTestSuiteIT. classMethod
ExpertScriptClientYamlTestSuiteIT. classMethod
ExpertScriptClientYamlTestSuiteIT. classMethod
ExpertScriptClientYamlTestSuiteIT. test {yaml=script_expert_scoring/10_basic/Plugin loaded}
ExpertScriptClientYamlTestSuiteIT. test {yaml=script_expert_scoring/10_basic/Plugin loaded}
ExpertScriptClientYamlTestSuiteIT. test {yaml=script_expert_scoring/10_basic/Plugin loaded}
ExpertScriptClientYamlTestSuiteIT. test {yaml=script_expert_scoring/10_basic/Plugin loaded}

same errors and failures as above test

9) CustomSuggesterClientYamlTestSuiteIT. classMethod CustomSuggesterClientYamlTestSuiteIT. classMethod CustomSuggesterClientYamlTestSuiteIT. classMethod CustomSuggesterClientYamlTestSuiteIT. classMethod CustomSuggesterClientYamlTestSuiteIT. test {yaml=custom-suggester/10_basic/plugin loaded} CustomSuggesterClientYamlTestSuiteIT. test {yaml=custom-suggester/10_basic/plugin loaded} CustomSuggesterClientYamlTestSuiteIT. test {yaml=custom-suggester/10_basic/plugin loaded} CustomSuggesterClientYamlTestSuiteIT. test {yaml=custom-suggester/10_basic/plugin loaded}

Poojita-Raj commented 1 year ago

Need to backport changes needed for windows CI to run to both 1.x and 1.3 branch.

These are all the changes that have taken place -

[x] 1. https://github.com/opensearch-project/OpenSearch/pull/5077 - ResourceAwareTasksTests doesn't exist on 1.x and 1.3 branches so this change doesn't need to be included.
[x] 2. https://github.com/opensearch-project/OpenSearch/pull/5023 - This fix has 2 open backport PRs for 1.x and 1.3.
[x] 3. https://github.com/opensearch-project/OpenSearch/pull/4941 - This fix has already been backported to 1.3, there is an open PR for 1.x backport as well.
[x] 4. https://github.com/opensearch-project/OpenSearch/pull/4949 - This is to revert Dan's PR on admin access, but it was never included in 1.3 and 1.x branches so doesn't need to be backported.

peternied commented 1 year ago

@Poojita-Raj Is there an effort to get OpenSearch 1.3 working on Windows? As far as I know, support for Windows in only for OpenSearch 2.4+

peterzhuamazon commented 1 year ago

Hi, just want to make sure this issue is closed, if all the tasks are completed.

Thanks.

dblock commented 1 year ago

Related issue is https://github.com/opensearch-project/OpenSearch/issues/1975. If we break up gradle check we can decide which sets need to run on all platforms as part of CI, and which need to be run on cron.

mch2 commented 1 year ago

Added the following tasks still remaining.

[ ] Fix gradle check on 1.3/1.x
[ ] Add GH workflow for windows to gate PRs. Add unit tests only.

peterzhuamazon commented 1 year ago

1.x: https://build.ci.opensearch.org/job/gradle-check/7060/ 1.3: https://build.ci.opensearch.org/job/gradle-check/7061/ Both windows have some errors here.

peterzhuamazon commented 1 year ago

windows: main: https://build.ci.opensearch.org/job/gradle-check/7152/ 2.x: https://build.ci.opensearch.org/job/gradle-check/7153/

CEHENKLE commented 1 year ago

@mch2 @peterzhuamazon @Poojita-Raj @minalsha

So what do we need to close this issue out with confidence?

mch2 commented 1 year ago

This issue specifically is around a PR gate on windows, we have additional work required to get on-merge/nightlies to green.

The question is what to run and where.

What to run? Ideally the entire check, but we've seen instability with our nightly runs when running the check. I've done some experimenting in breaking this up with existing gradle tasks here for 2.5. I suggest we start with :server:test as a PR gate and the rest on-merge/nightly.

Where to run?

Use GH runners (similar to our existing precommit wf) and run very lightweight tests. These runners are not powerful enough to run :server:test, so we will need to further break apart the test tasks to find an appropriate subset of tests to run that it can handle.
Use windows runners in Jenkins to run :server:test on-PR. From what I understand from @peterzhuamazon this will have significant (~25+ mins) preparation time right now on our CI before any job starts executing. I would prefer to go this route and work on cutting down that prep time.

minalsha commented 1 year ago

@mch2 what is needed to close this issue? Is it following pending workitems in the issue description?

mch2 commented 1 year ago

@mch2 what is needed to close this issue? Is it following pending workitems in the issue description?

@minalsha yes, I just cleaned up the issue description. The first step is getting stability in our nightly runs both from jenkins & the check and trigger them from GH. Then we will be able to transition to on PR checks. The previous issues we had about running a subset on GH i've closed as not enough of the gradle tasks would succeed to be worthwhile.

opensearch-project / OpenSearch