[Bug]: Windows k-NN plugin test fail on the cleanup phase

[jordarlu]

Describe the bug

k-NN team was doing sanity test on v1.3.9, and found the following :

Administrator@EC2AMAZ-2BOO3SA MINGW64 ~/Documents/k-NN (1.3)
$ ./gradlew.bat :integTestRemote -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="integTest-0" -Dhttps=false -PnumNodes=1
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 6.6.1
  OS Info               : Windows Server 2019 10.0 (amd64)
  JDK Version           : 11 (JDK)
  JAVA_HOME             : C:\Users\Administrator\scoop\apps\temurin11-jdk\11.0.17-8
  Random Testing Seed   : BB6E14DCF066E8CE
  In FIPS 140 mode      : false
=======================================
> Task :integTestRemote FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':integTestRemote'.
> java.io.IOException: Unable to delete directory 'C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0'
    Failed to delete some children. This might happen because a process has files open or has its working directory set in the target directory.
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\data\nodes\0\node.lock
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\data\nodes\0\_state\write.lock
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\data\nodes\0\_state
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\data\nodes\0
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\data\nodes
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\data
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\distro\1.3.9-INTEG_TEST\lib\HdrHistogram-2.1.9.jar
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\distro\1.3.9-INTEG_TEST\lib\hppc-0.8.1.jar
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\distro\1.3.9-INTEG_TEST\lib\jackson-core-2.14.2.jar
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\distro\1.3.9-INTEG_TEST\lib\jackson-dataformat-cbor-2.14.2.jar
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\distro\1.3.9-INTEG_TEST\lib\jackson-dataformat-smile-2.14.2.jar
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\distro\1.3.9-INTEG_TEST\lib\jackson-dataformat-yaml-2.14.2.jar
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\distro\1.3.9-INTEG_TEST\lib\java-version-checker-1.3.9-SNAPSHOT.jar
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\distro\1.3.9-INTEG_TEST\lib\jna-5.5.0.jar
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\distro\1.3.9-INTEG_TEST\lib\joda-time-2.12.2.jar
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\distro\1.3.9-INTEG_TEST\lib\jopt-simple-5.0.4.jar
    - and more ...
    New files were found. This might happen because a process is still writing to the target directory.
    - C:\Users\Administrator\Documents\k-NN\build\testclusters\integTestRemote-0\logs\opensearch.stdout.log
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org/
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 2s
7 actionable tasks: 1 executed, 6 up-to-date

According to @naveentatikonda , this issue has been seen on a few previous release such as 1.3.7 ...

To reproduce

./gradlew.bat :integTestRemote -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="integTest-0" -Dhttps=false -PnumNodes=1

Expected behavior

No response

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[bbarani]

@naveentatikonda Whats the plan to fix this issue?

[jordarlu]

Hi, @bbarani .. my understanding is this issue is not verion specific bug ( and it was observed on earier release Windows test as well ), but probably env settings bug that our team can triage next week. @rishabh6788 and @peterzhuamazon may have more detail on this finding.. thanks.

[naveentatikonda]

@bbarani as @jordarlu mentioned this is not something which is found in 1.3.9 testing. It was known before and I believe this is not related to k-NN plugin. It is a generic issue which is also faced by other plugin teams.

[Divyaasm]

Adding https://github.com/opensearch-project/opensearch-build/issues/3196 for tracking the issue