search

jetoile / hadoop-unit

Hadoop-Unit is a project which allow testing projects which need hadoop ecosysteme like kafka, solr, hdfs, hive, hbase, ...
https://blog.jetoile.fr/hadoop-unit/
Apache License 2.0
51 stars 15 forks source link

java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addDeprecations #92

Closed LvffY closed 5 years ago

LvffY commented 5 years ago

Hello Jetoile,

First of all, I'ld like to thank you for all this great work ! :)

I'm currently trying to improve our practices on my project and I try to introduce Oozie integration tests.

But for now I'm stuck in a "classNotfoundError".

For more details :

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>test</groupId>
    <artifactId>test</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <junit.version>5.4.0</junit.version>
        <junit.platform.version>1.3.2</junit.platform.version>
        <hadoop.version>2.7.3</hadoop.version>
        <hadoop-unit.version>3.4</hadoop-unit.version>
    </properties>

    <dependencies>

        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-runner</artifactId>
            <version>${junit.platform.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-params</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.oozie/oozie-client -->
        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-client</artifactId>
            <version>4.2.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-core -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>1.2.1</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-azure</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>fr.jetoile.hadoop</groupId>
            <artifactId>hadoop-unit-client-hdfs</artifactId>
            <version>${hadoop-unit.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>fr.jetoile.hadoop</groupId>
            <artifactId>hadoop-unit-client-hive</artifactId>
            <version>${hadoop-unit.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>fr.jetoile.hadoop</groupId>
            <artifactId>hadoop-unit-commons</artifactId>
            <version>${hadoop-unit.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>fr.jetoile.hadoop</groupId>
            <artifactId>hadoop-unit-oozie</artifactId>
            <version>${hadoop-unit.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.junit.platform/junit-platform-runner -->
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-runner</artifactId>
            <version>${junit.platform.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-params</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <excludes>
                        <exclude>**/*IntegrationTest.java</exclude>
                    </excludes>
                </configuration>
                <executions>
                    <execution>
                        <id>integration-test</id>
                        <goals>
                            <goal>test</goal>
                        </goals>
                        <phase>integration-test</phase>
                        <configuration>
                            <excludes>
                                <exclude>none</exclude>
                            </excludes>
                            <includes>
                                <include>**/*IntegrationTest.java</include>
                            </includes>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <artifactId>hadoop-unit-maven-plugin</artifactId>
                <groupId>fr.jetoile.hadoop</groupId>
                <version>${hadoop-unit.version}</version>
                <executions>
                    <execution>
                        <id>start</id>
                        <goals>
                            <goal>embedded-start</goal>
                        </goals>
                        <phase>pre-integration-test</phase>
                    </execution>
                    <execution>
                        <id>stop</id>
                        <goals>
                            <goal>embedded-stop</goal>
                        </goals>
                        <phase>post-integration-test</phase>
                    </execution>
                </executions>
                <configuration>
                    <components>
                        <componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
                            <componentName>HDFS</componentName>
                            <artifactId>hadoop-unit-hdfs</artifactId>
                            <groupId>fr.jetoile.hadoop</groupId>
                            <version>${hadoop-unit.version}</version>
                            <mainClass>fr.jetoile.hadoopunit.component.HdfsBootstrap</mainClass>
                        </componentArtifact>
                        <componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
                            <componentName>HIVEMETA</componentName>
                            <artifactId>hadoop-unit-hive</artifactId>
                            <groupId>fr.jetoile.hadoop</groupId>
                            <version>${hadoop-unit.version}</version>
                            <mainClass>fr.jetoile.hadoopunit.component.HiveMetastoreBootstrap</mainClass>
                        </componentArtifact>
                        <componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
                            <componentName>HIVESERVER2</componentName>
                            <artifactId>hadoop-unit-hive</artifactId>
                            <groupId>fr.jetoile.hadoop</groupId>
                            <version>${hadoop-unit.version}</version>
                            <mainClass>fr.jetoile.hadoopunit.component.HiveServer2Bootstrap</mainClass>
                        </componentArtifact>
                        <componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
                            <componentName>OOZIE</componentName>
                            <artifactId>hadoop-unit-oozie</artifactId>
                            <groupId>fr.jetoile.hadoop</groupId>
                            <version>${hadoop-unit.version}</version>
                            <mainClass>fr.jetoile.hadoopunit.component.OozieBootstrap</mainClass>
                        </componentArtifact>
                    </components>

                </configuration>

            </plugin>
        </plugins>
    </build>

</project>

Test class :

import fr.jetoile.hadoopunit.HadoopBootstrap;
import org.apache.commons.configuration.Configuration;
import org.apache.commons.configuration.ConfigurationException;
import org.apache.commons.configuration.PropertiesConfiguration;
import org.junit.Assert;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

class ScheduleIntegrationTest {

    static private Configuration configuration ;

    @BeforeEach
    void setUp() throws ConfigurationException {
        configuration = new PropertiesConfiguration("hadoop-unit-default.properties") ;
        HadoopBootstrap.INSTANCE.startAll();
    }

    @AfterEach
    void tearDown() {
        HadoopBootstrap.INSTANCE.stopAll();
    }

    @Test
    public void aVoid(){
        Assert.assertTrue(true);
    }
}

Stack trace :

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/D:/Profiles/aberges/.m2/repository/org/slf4j/slf4j-simple/1.6.6/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/D:/Profiles/aberges/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/D:/Profiles/aberges/.m2/repository/com/github/sakserv/property-parser/0.0.2/property-parser-0.0.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
298 [main] INFO fr.jetoile.hadoopunit.HadoopUtils - Writing winutils.exe to: C:\Hadoop\hadoop-2.7.3\bin\winutils.exe
Writing dll to: d:\Profiles\aberges\AppData\Local\Temp\ef0c8b46-970f-4630-8065-5d1e6d9a57e1\hadoop.dll
Writing dll to: d:\Profiles\aberges\AppData\Local\Temp\53431b51-6a46-4c39-b6ae-18ca86c73415\hdfs.dll
Writing dll to: d:\Profiles\aberges\AppData\Local\Temp\1d86e3ca-8070-4699-b6d4-ba01c5c61c0d\libwinutils.lib
2466 [main] INFO fr.jetoile.hadoopunit.component.HdfsBootstrap - fr.jetoile.hadoopunit.component.HdfsBootstrap is starting
19/05/18 12:14:15 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively

java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDelta;)V

    at org.apache.hadoop.hdfs.HdfsConfiguration.addDeprecatedKeys(HdfsConfiguration.java:66)
    at org.apache.hadoop.hdfs.HdfsConfiguration.<clinit>(HdfsConfiguration.java:31)
    at fr.jetoile.hadoopunit.component.HdfsBootstrap.build(HdfsBootstrap.java:92)
    at fr.jetoile.hadoopunit.component.HdfsBootstrap.start(HdfsBootstrap.java:181)
    at fr.jetoile.hadoopunit.HadoopBootstrap.startService(HadoopBootstrap.java:146)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at fr.jetoile.hadoopunit.HadoopBootstrap.internalStart(HadoopBootstrap.java:134)
    at fr.jetoile.hadoopunit.HadoopBootstrap.startAll(HadoopBootstrap.java:112)
    at ScheduleIntegrationTest.setUp(ScheduleIntegrationTest.java:17)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:628)
    at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:117)
    at org.junit.jupiter.engine.descriptor.ClassTestDescriptor.invokeMethodInExtensionContext(ClassTestDescriptor.java:439)
    at org.junit.jupiter.engine.descriptor.ClassTestDescriptor.lambda$synthesizeBeforeEachMethodAdapter$15(ClassTestDescriptor.java:427)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeEachMethods$3(TestMethodTestDescriptor.java:149)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeMethodsOrCallbacksUntilExceptionOccurs(TestMethodTestDescriptor.java:169)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeEachMethods(TestMethodTestDescriptor.java:148)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:123)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
    at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
    at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
    at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:69)
    at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

2661 [Thread-1] INFO fr.jetoile.hadoopunit.HadoopBootstrap - All services are going to be stopped

Process finished with exit code -1

I dont know if you need more details, but if so, don't hesitate !

jetoile commented 5 years ago

Hello Lvffy,

Thanks for your feedback.

In Hadoop Unit you have different modes:

You have to choose your mode but do not use both of them.

So, you should remove:

        <dependency>
            <groupId>fr.jetoile.hadoop</groupId>
            <artifactId>hadoop-unit-commons</artifactId>
            <version>${hadoop-unit.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>fr.jetoile.hadoop</groupId>
            <artifactId>hadoop-unit-oozie</artifactId>
            <version>${hadoop-unit.version}</version>
            <scope>test</scope>
        </dependency>

and 

    @BeforeEach
    void setUp() throws ConfigurationException {
        configuration = new PropertiesConfiguration("hadoop-unit-default.properties") ;
        HadoopBootstrap.INSTANCE.startAll();
    }

    @AfterEach
    void tearDown() {
        HadoopBootstrap.INSTANCE.stopAll();
    }

and just keep the plugin mode.

Does it make sense?

LvffY commented 5 years ago

Thanks for your quick answer !

Yes it makes sense.

When I apply your advices on my codes, my test passed ... But I don't see any actions from your plugin ... Do I need to execute an Hadoop command for starting your plugin ?

jetoile commented 5 years ago

Since the plugin is set on the pre-integration test maven's phase, you can run the command mvn verify mvn integration-test or mvn install.

Note that you should override settings for oozie with something like:

<componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
    <componentName>OOZIE</componentName>
    <artifactId>hadoop-unit-solrcloud</artifactId>
    <groupId>fr.jetoile.hadoop</groupId>
    <version>${hadoop-unit.version}</version>
    <mainClass>fr.jetoile.hadoopunit.component.OozieBootstrap</mainClass>
    <properties>
        <your-properties-key>your-value</your-properties-key>
    </properties>
</componentArtifact>

where properties are set by default with:

oozie.tmp.dir=/tmp/oozie_tmp
oozie.test.dir=/tmp/embedded_oozie
oozie.home.dir=/tmp/oozie_home
oozie.username=blah
oozie.groupname=testgroup
oozie.hdfs.share.lib.dir=/tmp/share_lib
oozie.share.lib.create=true
oozie.local.share.lib.cache.dir=/tmp/share_lib_cache
oozie.purge.local.share.lib.cache=false
oozie.sharelib.path=~/github
oozie.sharelib.name=oozie-4.2.0.2.6.3.2-235-distro.tar.gz
oozie.port=20113
oozie.host=127.0.0.1
oozie.sharelib.component=OOZIE,MAPREDUCE_STREAMING,SPARK
LvffY commented 5 years ago

Okay, thanks for the tips.

When seeing the differents properties name, I guess I need to download the oozie sharelib ? Does a maven downloading can be enough ?

Are these configurations related to the project or the computer ? i.e If I put oozie.tmp.dir=/tmp/oozie will the path be in my maven project or not ?

Final question : Do you have any docs on theses configurations in addition of Oozie docs ?

jetoile commented 5 years ago

If you do not need shared libs, you can override the property oozie.share.lib.create to false.

If you need it, you have to download it first and to set the properties oozie.sharelib.path and oozie.sharelib.name. (see here)

The property oozieTmpDir is the folder where oozie's shared lib will be unzip before being copy into hdfs.

Finally, note that hadoop unit use hadoop-mini-cluster so you can refer to this project to get informations.

Hope this helps.

LvffY commented 5 years ago

I do need sharelibs.

Note : In your blog your link to download Oozie sharelib seems to be dead

For my project I'm using Azure HDInsight clusters. In these installation, in each Oozie sharelib (oozie, hcatalog, Hive etc ...) I can find many oozie-sharelib*.jar. Do i need to download all of them (or at least each one I need in my test environment) ?

jetoile commented 5 years ago

Hello Lvffy,

Thanks for the dead link. It is solved now and the link is: https://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.3.0/tars/oozie/oozie-4.2.0.2.6.3.0-235-distro.tar.gz

The link above have all dependencies.

LvffY commented 5 years ago

Thanks for the fix !

It seems that your link point to a sharelib for HDP 2.6.3. As I want to use hadoop-unit 3.4 and this version mocks HDP 2.6.5, is there any risk and/or limitations regarding differents versiosns ?

LvffY commented 5 years ago

I think i have setup all the way you advise me.

Now when I run my maven command, I see the differents server starting ... Except for Oozie.

To compare from my first message :

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>peps-scala</artifactId>
        <groupId>com.sncf.pf.apo.ssg.cdcbibigdata</groupId>
        <version>0.0.1-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>schedule</artifactId>

    <properties>
        <hadoop.version>2.7.3.2.6.3.40-13</hadoop.version>
        <hadoop-unit.version>3.4</hadoop-unit.version>
        <junit.version>5.4.0</junit.version>
        <junit.platform.version>1.3.2</junit.platform.version>
    </properties>

    <dependencies>

        <!-- https://mvnrepository.com/artifact/org.apache.oozie/oozie-client -->
        <dependency>
            <groupId>org.apache.oozie</groupId>
            <artifactId>oozie-client</artifactId>
            <version>4.2.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure-datalake -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-azure-datalake</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-azure</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.junit.platform/junit-platform-runner -->
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-runner</artifactId>
            <version>${junit.platform.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-params</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <excludes>
                        <exclude>**/*IntegrationTest.java</exclude>
                    </excludes>
                </configuration>
                <executions>
                    <execution>
                        <id>integration-test</id>
                        <goals>
                            <goal>test</goal>
                        </goals>
                        <phase>integration-test</phase>
                        <configuration>
                            <excludes>
                                <exclude>none</exclude>
                            </excludes>
                            <includes>
                                <include>**/*IntegrationTest.java</include>
                            </includes>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.1</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <artifactSet>
                                <includes>
                                    <include>org.apache.oozie:oozie-client</include>
                                    <include>com.googlecode.json-simple:json-simple</include>
                                </includes>
                            </artifactSet>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <artifactId>hadoop-unit-maven-plugin</artifactId>
                <groupId>fr.jetoile.hadoop</groupId>
                <version>${hadoop-unit.version}</version>
                <executions>
                    <execution>
                        <id>start</id>
                        <goals>
                            <goal>embedded-start</goal>
                        </goals>
                        <phase>pre-integration-test</phase>
                    </execution>
                    <execution>
                        <id>stop</id>
                        <goals>
                            <goal>embedded-stop</goal>
                        </goals>
                        <phase>post-integration-test</phase>
                    </execution>
                </executions>
                <configuration>
                    <components>
                        <componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
                            <componentName>HDFS</componentName>
                            <artifactId>hadoop-unit-hdfs</artifactId>
                            <groupId>fr.jetoile.hadoop</groupId>
                            <version>${hadoop-unit.version}</version>
                            <mainClass>fr.jetoile.hadoopunit.component.HdfsBootstrap</mainClass>
                        </componentArtifact>
                        <componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
                            <componentName>HIVEMETA</componentName>
                            <artifactId>hadoop-unit-hive</artifactId>
                            <groupId>fr.jetoile.hadoop</groupId>
                            <version>${hadoop-unit.version}</version>
                            <mainClass>fr.jetoile.hadoopunit.component.HiveMetastoreBootstrap</mainClass>
                        </componentArtifact>
                        <componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
                            <componentName>HIVESERVER2</componentName>
                            <artifactId>hadoop-unit-hive</artifactId>
                            <groupId>fr.jetoile.hadoop</groupId>
                            <version>${hadoop-unit.version}</version>
                            <mainClass>fr.jetoile.hadoopunit.component.HiveServer2Bootstrap</mainClass>
                        </componentArtifact>
                        <componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
                            <componentName>OOZIE</componentName>
                            <artifactId>hadoop-unit-oozie</artifactId>
                            <groupId>fr.jetoile.hadoop</groupId>
                            <version>${hadoop-unit.version}</version>
                            <mainClass>fr.jetoile.hadoopunit.component.OozieBootstrap</mainClass>
                            <properties>
                                <oozie.tmp.dir>/tmp/oozie_tmp</oozie.tmp.dir>
                                <oozie.test.dir>/tmp/embedded_oozie</oozie.test.dir>
                                <oozie.home.dir>/tmp/oozie_home</oozie.home.dir>
                                <oozie.username>${user.name}</oozie.username>
                                <oozie.groupname>${user.name}</oozie.groupname>
                                <oozie.hdfs.share.lib.dir>/tmp/share_lib</oozie.hdfs.share.lib.dir>
                                <oozie.share.lib.create>true</oozie.share.lib.create>
                                <oozie.local.share.lib.cache.dir>/tmp/share_lib_cache</oozie.local.share.lib.cache.dir>
                                <oozie.purge.local.share.lib.cache>false</oozie.purge.local.share.lib.cache>
                                <oozie.sharelib.path>/tmp</oozie.sharelib.path>
                                <oozie.sharelib.name>oozie-4.2.0.2.6.3.0-235-distro.tar.gz</oozie.sharelib.name>
                                <oozie.port>20113</oozie.port>
                                <oozie.host>127.0.0.1</oozie.host>
                                <oozie.sharelib.component>OOZIE,SHELL,SPARK,HIVE2,
HCATALOG</oozie.sharelib.component>
                            </properties>
                        </componentArtifact>
                    </components>

                </configuration>

            </plugin>
        </plugins>
    </build>
</project>

Test class

package com.sncf.pf.apo.ssg.cdcbibigdata.peps.notscala;

import org.junit.Assert;
import org.junit.jupiter.api.Test;

class ScheduleIntegrationTest {

    @Test
    public void aVoid(){
        Assert.assertTrue(true);
    }
}

Stack trace

After running mvn integration-test I get the following stack trace log.txt

Any clue on the problem ?

jetoile commented 5 years ago

You're right. Here is the link to get the oozie's sharelib for 2.6.5: http://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.5.0/tars/oozie/oozie-4.2.0.2.6.5.0-292-distro.tar.gz

For your error, SHELL and HCATALOG is not a supported value. See hadoop-mini-cluster

Here are the corrects values:

SHELL is already included.

Hope this helps.

LvffY commented 5 years ago

Seems to work, thanks a lot !

Another question, a way ahead from the issue itself : On my project, we would like to use Spark2 and Oozie. There are know issues that make it impossible from scratch.

Thanks to Hortonworks, we have an official workaround. Unfortunately, this workaround works only with an already running Oozie e.g after server starting.

Do you think that if I rebuild Oozie distro in replacing spark sharelib by spark2 sharelib this could work ? Is it better to ask this question on hadoop-mini-cluster github page ?

LvffY commented 5 years ago

Another question : In your OozieBootstrapTest you don't use the maven plugin, you start the HadoopBootstrap "manually". In the case we use the plugin, do you have any programatical way to access informations from the differents servers (i.e HDFS FileSystem, Oozie URI, etc) ?

jetoile commented 5 years ago

Hum... to setup spark2 for oozie, I think that you can setup the sharelib by code if you get what you need from a clean hdp setup (/usr/hdp/current/spark2-client/jars/* and others stuff). But I am not sure it will succeed...

But are you sure you want to test your job through oozie which run on yarn? The spark local mode is not enough?

OozieBootstrapTest use the mode where start oozie into the same context as your test but with the plugin mode or the standalone mode, you can access oozie and other components from outside.

By default:

You have some samples here

Hope this helps

LvffY commented 5 years ago

Right now, I'm on a POC stage without any time constraints so I would like to make the exercice until the end. By this, I mean try things even if we don't use them every day or even once.

For the Spark2 configuration : right now we're unable to make this workaround on our integration environment so this is not the priority, but thanks for the answer 😄

To answer your question : I already tested all parts of my code in Spark local mode. Now I would want to test my Oozie Workflow. But I share your concerns : Is it really necessary to test both Spark local and integration with Oozie ... I'm not really theorical, so I need to get my hands dirty to really understand what's going on and what is useful. For now, I'm not sure we're gonna use this Oozie testing all the time, may be just for some specifics use cases and/or tricky workflows)

Thanks for the adresses, I thought some variables would have been initialized by the framework to get theses independently from my installation. In my case, it doesn't really matter because all my configurations are by default.

jetoile commented 5 years ago

Ok thanks for your reply.

Can I close the issue?

Feel free to post PR if docs or other stuff are wrong :P

LvffY commented 5 years ago

Ok, thanks.

I close the issue, don't worry 😄