search

gchq / gaffer-tools

gaffer-tools is deprecated. Use https://github.com/gchq/gafferpy instead
Apache License 2.0
50 stars 29 forks source link

MiniAccumuloCluster fails tests on Windows #935

Closed t92549 closed 2 years ago

t92549 commented 3 years ago

Found: https://github.com/gchq/Gaffer/issues/2509

It seems that there is an issue with file permissions when using the MiniAccumuloCluster helper package on Windows. Full stack trace from original issue:

[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] gaffer-tools                                                       [pom]
[INFO] ui                                                                 [war]
[INFO] mini-accumulo-cluster                                              [jar]
[INFO] random-element-generation                                          [jar]
[INFO] performance-testing                                                [pom]
[INFO] performance-testing-core                                           [jar]
[INFO] performance-testing-accumulo-store                                 [jar]
[INFO] performance-testing-aws                                            [jar]
[INFO] schema-builder                                                     [pom]
[INFO] schema-builder-rest                                                [war]
[INFO] slider                                                             [jar]
[INFO] analytics                                                          [pom]
[INFO] analytics-functions                                                [jar]
[INFO] analytics-operations                                               [jar]
[INFO] analytics-rest                                                     [war]
[INFO]
[INFO] ------------------< uk.gov.gchq.gaffer:gaffer-tools >-------------------
[INFO] Building gaffer-tools 1.18.1-SNAPSHOT                             [1/15]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-checkstyle-plugin:2.17:check (validate) @ gaffer-tools ---
[INFO] Starting audit...
Audit done.
[INFO]
[INFO] >>> findbugs-maven-plugin:3.0.5:check (analyze-compile) > :findbugs @ gaffer-tools >>>
[INFO]
[INFO] --- findbugs-maven-plugin:3.0.5:findbugs (findbugs) @ gaffer-tools ---
[INFO]
[INFO] <<< findbugs-maven-plugin:3.0.5:check (analyze-compile) < :findbugs @ gaffer-tools <<<
[INFO]
[INFO]
[INFO] --- findbugs-maven-plugin:3.0.5:check (analyze-compile) @ gaffer-tools ---
[INFO]
[INFO] --- maven-jar-plugin:2.4:test-jar (default) @ gaffer-tools ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO]
[INFO] --- maven-source-plugin:2.4:jar-no-fork (attach-sources) @ gaffer-tools ---
[INFO]
[INFO] --- maven-javadoc-plugin:2.10.3:jar (attach-javadocs) @ gaffer-tools ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable package
[INFO]
[INFO] -----------------------< uk.gov.gchq.gaffer:ui >------------------------
[INFO] Building ui 1.18.1-SNAPSHOT                                       [2/15]
[INFO] --------------------------------[ war ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ ui ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory C:\Users\Asus\Desktop\gaffer-tools-develop\ui\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ ui ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ ui ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ ui ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ ui ---
[INFO]
[INFO] --- maven-checkstyle-plugin:2.17:check (validate) @ ui ---
[INFO] Starting audit...
Audit done.
[INFO]
[INFO] >>> findbugs-maven-plugin:3.0.5:check (analyze-compile) > :findbugs @ ui >>>
[INFO]
[INFO] --- findbugs-maven-plugin:3.0.5:findbugs (findbugs) @ ui ---
[INFO]
[INFO] <<< findbugs-maven-plugin:3.0.5:check (analyze-compile) < :findbugs @ ui <<<
[INFO]
[INFO]
[INFO] --- findbugs-maven-plugin:3.0.5:check (analyze-compile) @ ui ---
[INFO]
[INFO] --- maven-war-plugin:3.0.0:war (default-war) @ ui ---
[INFO] Packaging webapp
[INFO] Assembling webapp [ui] in [C:\Users\Asus\Desktop\gaffer-tools-develop\ui\target\ui-1.18.1-SNAPSHOT]
[INFO] Processing war project
[INFO] Copying webapp webResources [C:\Users\Asus\Desktop\gaffer-tools-develop\ui\dependencies] to [C:\Users\Asus\Desktop\gaffer-tools-develop\ui\target\ui-1.18.1-SNAPSHOT]
[INFO] Copying webapp resources [C:\Users\Asus\Desktop\gaffer-tools-develop\ui\src\main\webapp]
[INFO] Webapp assembled in [5053 msecs]
[INFO] Building war: C:\Users\Asus\Desktop\gaffer-tools-develop\ui\target\ui-1.18.1-SNAPSHOT.war
[INFO]
[INFO] --- maven-jar-plugin:2.4:test-jar (default) @ ui ---
[INFO] Building jar: C:\Users\Asus\Desktop\gaffer-tools-develop\ui\target\ui-1.18.1-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-source-plugin:2.4:jar-no-fork (attach-sources) @ ui ---
[INFO] No sources in project. Archive not created.
[INFO]
[INFO] --- maven-javadoc-plugin:2.10.3:jar (attach-javadocs) @ ui ---
[INFO]
[INFO] --------------< uk.gov.gchq.gaffer:mini-accumulo-cluster >--------------
[INFO] Building mini-accumulo-cluster 1.18.1-SNAPSHOT                    [3/15]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ mini-accumulo-cluster ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ mini-accumulo-cluster ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ mini-accumulo-cluster ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ mini-accumulo-cluster ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ mini-accumulo-cluster ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running uk.gov.gchq.gaffer.miniaccumulocluster.MiniAccumuloClusterControllerTest
gaffer.miniaccumulocluster.MiniAccumuloClusterController ERROR  - Failed to delete old directory
java.io.IOException: Unable to delete file: miniAccumuloCluster\zookeeper
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1338)
        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:334)
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1121)
        at uk.gov.gchq.gaffer.miniaccumulocluster.MiniAccumuloClusterController.createDirectory(MiniAccumuloClusterController.java:269)
        at uk.gov.gchq.gaffer.miniaccumulocluster.MiniAccumuloClusterController.start(MiniAccumuloClusterController.java:121)
        at uk.gov.gchq.gaffer.miniaccumulocluster.MiniAccumuloClusterControllerTest.lambda$shouldStartAndStopCluster$0(MiniAccumuloClusterControllerTest.java:45)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: miniAccumuloCluster\zookeeper\version-2\log.1: Impossibile accedere al file. Il file è utilizzato da un altro processo.

        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
        at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
        at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
        at java.nio.file.Files.deleteIfExists(Files.java:1165)
        at org.apache.commons.io.file.DeletingPathVisitor.visitFile(DeletingPathVisitor.java:99)
        at org.apache.commons.io.file.DeletingPathVisitor.visitFile(DeletingPathVisitor.java:35)
        at java.nio.file.Files.walkFileTree(Files.java:2670)
        at java.nio.file.Files.walkFileTree(Files.java:2742)
        at org.apache.commons.io.file.PathUtils.visitFileTree(PathUtils.java:535)
        at org.apache.commons.io.file.PathUtils.deleteDirectory(PathUtils.java:265)
        at org.apache.commons.io.file.PathUtils.delete(PathUtils.java:254)
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1336)
        ... 6 more
gaffer.miniaccumulocluster.MiniAccumuloClusterController ERROR  - Failed to create temp dir
java.nio.file.FileAlreadyExistsException: miniAccumuloCluster
        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:81)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
        at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:504)
        at java.nio.file.Files.createDirectory(Files.java:674)
        at uk.gov.gchq.gaffer.miniaccumulocluster.MiniAccumuloClusterController.createDirectory(MiniAccumuloClusterController.java:279)
        at uk.gov.gchq.gaffer.miniaccumulocluster.MiniAccumuloClusterController.start(MiniAccumuloClusterController.java:121)
        at uk.gov.gchq.gaffer.miniaccumulocluster.MiniAccumuloClusterControllerTest.lambda$shouldStartAndStopCluster$0(MiniAccumuloClusterControllerTest.java:45)
        at java.lang.Thread.run(Thread.java:748)
java.io.IOException: Unable to delete file: miniAccumuloCluster\zookeeper
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1338)
        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:334)
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1121)
        at uk.gov.gchq.gaffer.miniaccumulocluster.MiniAccumuloClusterController.stop(MiniAccumuloClusterController.java:141)
        at uk.gov.gchq.gaffer.miniaccumulocluster.MiniAccumuloClusterControllerTest.shouldStartAndStopCluster(MiniAccumuloClusterControllerTest.java:66)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
        at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
        at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
        at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.Iterator.forEachRemaining(Iterator.java:116)
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
        at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:248)
        at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$5(DefaultLauncher.java:211)
        at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:226)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:199)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:132)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:128)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
        at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
        at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Caused by: java.nio.file.FileSystemException: miniAccumuloCluster\zookeeper\version-2\log.1: Impossibile accedere al file. Il file è utilizzato da un altro processo.

        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
        at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
        at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
        at java.nio.file.Files.deleteIfExists(Files.java:1165)
        at org.apache.commons.io.file.DeletingPathVisitor.visitFile(DeletingPathVisitor.java:99)
        at org.apache.commons.io.file.DeletingPathVisitor.visitFile(DeletingPathVisitor.java:35)
        at java.nio.file.Files.walkFileTree(Files.java:2670)
        at java.nio.file.Files.walkFileTree(Files.java:2742)
        at org.apache.commons.io.file.PathUtils.visitFileTree(PathUtils.java:535)
        at org.apache.commons.io.file.PathUtils.deleteDirectory(PathUtils.java:265)
        at org.apache.commons.io.file.PathUtils.delete(PathUtils.java:254)
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1336)
        ... 51 more
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.564 s <<< FAILURE! - in uk.gov.gchq.gaffer.miniaccumulocluster.MiniAccumuloClusterControllerTest
[ERROR] uk.gov.gchq.gaffer.miniaccumulocluster.MiniAccumuloClusterControllerTest.shouldStartAndStopCluster  Time elapsed: 5.535 s  <<< FAILURE!
java.lang.AssertionError: Unable to create directory expected null, but was:<java.lang.RuntimeException: Unable to create directory>
        at uk.gov.gchq.gaffer.miniaccumulocluster.MiniAccumuloClusterControllerTest.shouldStartAndStopCluster(MiniAccumuloClusterControllerTest.java:61)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   MiniAccumuloClusterControllerTest.shouldStartAndStopCluster:61 Unable to create directory expected null, but was:<java.lang.RuntimeException: Unable to create directory>
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for gaffer-tools 1.18.1-SNAPSHOT:
[INFO]
[INFO] gaffer-tools ....................................... SUCCESS [  4.434 s]
[INFO] ui ................................................. SUCCESS [ 10.827 s]
[INFO] mini-accumulo-cluster .............................. FAILURE [  8.730 s]
[INFO] random-element-generation .......................... SKIPPED
[INFO] performance-testing ................................ SKIPPED
[INFO] performance-testing-core ........................... SKIPPED
[INFO] performance-testing-accumulo-store ................. SKIPPED
[INFO] performance-testing-aws ............................ SKIPPED
[INFO] schema-builder ..................................... SKIPPED
[INFO] schema-builder-rest ................................ SKIPPED
[INFO] slider ............................................. SKIPPED
[INFO] analytics .......................................... SKIPPED
[INFO] analytics-functions ................................ SKIPPED
[INFO] analytics-operations ............................... SKIPPED
[INFO] analytics-rest ..................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  24.410 s
[INFO] Finished at: 2021-09-19T18:46:20+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project mini-accumulo-cluster: There are test failures.
[ERROR]
[ERROR] Please refer to C:\Users\Asus\Desktop\gaffer-tools-develop\mini-accumulo-cluster\target\surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :mini-accumulo-cluster
GCHQDeveloper314 commented 2 years ago

This problem is likely similar to those in Gaffer #2717. The build output above (supplied by the original reporter) should be first be double checked against the latest version to confirm this problem is still present.

The build output shows a problem with deleting a file caused by a java.nio.file.FileSystemException of "Impossibile accedere al file. Il file è utilizzato da un altro processo". This is almost certainly the same as "The process cannot access the file because it is being used by another process" error which is seen in #2717.

GCHQDeveloper314 commented 2 years ago

This issue is caused by not having the Hadoop native libraries (specifically hadoop.dll) installed. With them the tests pass as expected. Fix in #1026

GCHQDeveloper314 commented 2 years ago

Done (#1026)