Closed LvffY closed 5 years ago
Hello Lvffy,
Thanks for your feedback.
In Hadoop Unit you have different modes:
You have to choose your mode but do not use both of them.
So, you should remove:
<dependency>
<groupId>fr.jetoile.hadoop</groupId>
<artifactId>hadoop-unit-commons</artifactId>
<version>${hadoop-unit.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>fr.jetoile.hadoop</groupId>
<artifactId>hadoop-unit-oozie</artifactId>
<version>${hadoop-unit.version}</version>
<scope>test</scope>
</dependency>
and
@BeforeEach
void setUp() throws ConfigurationException {
configuration = new PropertiesConfiguration("hadoop-unit-default.properties") ;
HadoopBootstrap.INSTANCE.startAll();
}
@AfterEach
void tearDown() {
HadoopBootstrap.INSTANCE.stopAll();
}
and just keep the plugin mode.
Does it make sense?
Thanks for your quick answer !
Yes it makes sense.
When I apply your advices on my codes, my test passed ... But I don't see any actions from your plugin ... Do I need to execute an Hadoop command for starting your plugin ?
Since the plugin is set on the pre-integration test maven's phase, you can run the command mvn verify
mvn integration-test
or mvn install
.
Note that you should override settings for oozie with something like:
<componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
<componentName>OOZIE</componentName>
<artifactId>hadoop-unit-solrcloud</artifactId>
<groupId>fr.jetoile.hadoop</groupId>
<version>${hadoop-unit.version}</version>
<mainClass>fr.jetoile.hadoopunit.component.OozieBootstrap</mainClass>
<properties>
<your-properties-key>your-value</your-properties-key>
</properties>
</componentArtifact>
where properties are set by default with:
oozie.tmp.dir=/tmp/oozie_tmp
oozie.test.dir=/tmp/embedded_oozie
oozie.home.dir=/tmp/oozie_home
oozie.username=blah
oozie.groupname=testgroup
oozie.hdfs.share.lib.dir=/tmp/share_lib
oozie.share.lib.create=true
oozie.local.share.lib.cache.dir=/tmp/share_lib_cache
oozie.purge.local.share.lib.cache=false
oozie.sharelib.path=~/github
oozie.sharelib.name=oozie-4.2.0.2.6.3.2-235-distro.tar.gz
oozie.port=20113
oozie.host=127.0.0.1
oozie.sharelib.component=OOZIE,MAPREDUCE_STREAMING,SPARK
Okay, thanks for the tips.
When seeing the differents properties name, I guess I need to download the oozie sharelib ? Does a maven downloading can be enough ?
Are these configurations related to the project or the computer ? i.e If I put oozie.tmp.dir=/tmp/oozie will the path be in my maven project or not ?
Final question : Do you have any docs on theses configurations in addition of Oozie docs ?
If you do not need shared libs, you can override the property oozie.share.lib.create
to false
.
If you need it, you have to download it first and to set the properties oozie.sharelib.path
and oozie.sharelib.name
. (see here)
The property oozieTmpDir
is the folder where oozie's shared lib will be unzip before being copy into hdfs.
Finally, note that hadoop unit use hadoop-mini-cluster so you can refer to this project to get informations.
Hope this helps.
I do need sharelibs.
Note : In your blog your link to download Oozie sharelib seems to be dead
For my project I'm using Azure HDInsight clusters. In these installation, in each Oozie sharelib (oozie, hcatalog, Hive etc ...) I can find many oozie-sharelib*.jar. Do i need to download all of them (or at least each one I need in my test environment) ?
Hello Lvffy,
Thanks for the dead link. It is solved now and the link is: https://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.3.0/tars/oozie/oozie-4.2.0.2.6.3.0-235-distro.tar.gz
The link above have all dependencies.
Thanks for the fix !
It seems that your link point to a sharelib for HDP 2.6.3. As I want to use hadoop-unit 3.4 and this version mocks HDP 2.6.5, is there any risk and/or limitations regarding differents versiosns ?
I think i have setup all the way you advise me.
Now when I run my maven command, I see the differents server starting ... Except for Oozie.
To compare from my first message :
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>peps-scala</artifactId>
<groupId>com.sncf.pf.apo.ssg.cdcbibigdata</groupId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>schedule</artifactId>
<properties>
<hadoop.version>2.7.3.2.6.3.40-13</hadoop.version>
<hadoop-unit.version>3.4</hadoop-unit.version>
<junit.version>5.4.0</junit.version>
<junit.platform.version>1.3.2</junit.platform.version>
</properties>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.oozie/oozie-client -->
<dependency>
<groupId>org.apache.oozie</groupId>
<artifactId>oozie-client</artifactId>
<version>4.2.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure-datalake -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-azure-datalake</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-azure</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.junit.platform/junit-platform-runner -->
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-runner</artifactId>
<version>${junit.platform.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludes>
<exclude>**/*IntegrationTest.java</exclude>
</excludes>
</configuration>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>test</goal>
</goals>
<phase>integration-test</phase>
<configuration>
<excludes>
<exclude>none</exclude>
</excludes>
<includes>
<include>**/*IntegrationTest.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<includes>
<include>org.apache.oozie:oozie-client</include>
<include>com.googlecode.json-simple:json-simple</include>
</includes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>hadoop-unit-maven-plugin</artifactId>
<groupId>fr.jetoile.hadoop</groupId>
<version>${hadoop-unit.version}</version>
<executions>
<execution>
<id>start</id>
<goals>
<goal>embedded-start</goal>
</goals>
<phase>pre-integration-test</phase>
</execution>
<execution>
<id>stop</id>
<goals>
<goal>embedded-stop</goal>
</goals>
<phase>post-integration-test</phase>
</execution>
</executions>
<configuration>
<components>
<componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
<componentName>HDFS</componentName>
<artifactId>hadoop-unit-hdfs</artifactId>
<groupId>fr.jetoile.hadoop</groupId>
<version>${hadoop-unit.version}</version>
<mainClass>fr.jetoile.hadoopunit.component.HdfsBootstrap</mainClass>
</componentArtifact>
<componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
<componentName>HIVEMETA</componentName>
<artifactId>hadoop-unit-hive</artifactId>
<groupId>fr.jetoile.hadoop</groupId>
<version>${hadoop-unit.version}</version>
<mainClass>fr.jetoile.hadoopunit.component.HiveMetastoreBootstrap</mainClass>
</componentArtifact>
<componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
<componentName>HIVESERVER2</componentName>
<artifactId>hadoop-unit-hive</artifactId>
<groupId>fr.jetoile.hadoop</groupId>
<version>${hadoop-unit.version}</version>
<mainClass>fr.jetoile.hadoopunit.component.HiveServer2Bootstrap</mainClass>
</componentArtifact>
<componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
<componentName>OOZIE</componentName>
<artifactId>hadoop-unit-oozie</artifactId>
<groupId>fr.jetoile.hadoop</groupId>
<version>${hadoop-unit.version}</version>
<mainClass>fr.jetoile.hadoopunit.component.OozieBootstrap</mainClass>
<properties>
<oozie.tmp.dir>/tmp/oozie_tmp</oozie.tmp.dir>
<oozie.test.dir>/tmp/embedded_oozie</oozie.test.dir>
<oozie.home.dir>/tmp/oozie_home</oozie.home.dir>
<oozie.username>${user.name}</oozie.username>
<oozie.groupname>${user.name}</oozie.groupname>
<oozie.hdfs.share.lib.dir>/tmp/share_lib</oozie.hdfs.share.lib.dir>
<oozie.share.lib.create>true</oozie.share.lib.create>
<oozie.local.share.lib.cache.dir>/tmp/share_lib_cache</oozie.local.share.lib.cache.dir>
<oozie.purge.local.share.lib.cache>false</oozie.purge.local.share.lib.cache>
<oozie.sharelib.path>/tmp</oozie.sharelib.path>
<oozie.sharelib.name>oozie-4.2.0.2.6.3.0-235-distro.tar.gz</oozie.sharelib.name>
<oozie.port>20113</oozie.port>
<oozie.host>127.0.0.1</oozie.host>
<oozie.sharelib.component>OOZIE,SHELL,SPARK,HIVE2,
HCATALOG</oozie.sharelib.component>
</properties>
</componentArtifact>
</components>
</configuration>
</plugin>
</plugins>
</build>
</project>
package com.sncf.pf.apo.ssg.cdcbibigdata.peps.notscala;
import org.junit.Assert;
import org.junit.jupiter.api.Test;
class ScheduleIntegrationTest {
@Test
public void aVoid(){
Assert.assertTrue(true);
}
}
After running mvn integration-test
I get the following stack trace
log.txt
Any clue on the problem ?
You're right. Here is the link to get the oozie's sharelib for 2.6.5: http://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.5.0/tars/oozie/oozie-4.2.0.2.6.5.0-292-distro.tar.gz
For your error, SHELL and HCATALOG is not a supported value. See hadoop-mini-cluster
Here are the corrects values:
SHELL is already included.
Hope this helps.
Seems to work, thanks a lot !
Another question, a way ahead from the issue itself : On my project, we would like to use Spark2 and Oozie. There are know issues that make it impossible from scratch.
Thanks to Hortonworks, we have an official workaround. Unfortunately, this workaround works only with an already running Oozie e.g after server starting.
Do you think that if I rebuild Oozie distro in replacing spark sharelib by spark2 sharelib this could work ? Is it better to ask this question on hadoop-mini-cluster github page ?
Another question : In your OozieBootstrapTest you don't use the maven plugin, you start the HadoopBootstrap "manually". In the case we use the plugin, do you have any programatical way to access informations from the differents servers (i.e HDFS FileSystem, Oozie URI, etc) ?
Hum... to setup spark2 for oozie, I think that you can setup the sharelib by code if you get what you need from a clean hdp setup (/usr/hdp/current/spark2-client/jars/* and others stuff). But I am not sure it will succeed...
But are you sure you want to test your job through oozie which run on yarn? The spark local mode is not enough?
OozieBootstrapTest use the mode where start oozie into the same context as your test but with the plugin mode or the standalone mode, you can access oozie and other components from outside.
By default:
You have some samples here
Hope this helps
Right now, I'm on a POC stage without any time constraints so I would like to make the exercice until the end. By this, I mean try things even if we don't use them every day or even once.
For the Spark2 configuration : right now we're unable to make this workaround on our integration environment so this is not the priority, but thanks for the answer 😄
To answer your question : I already tested all parts of my code in Spark local mode. Now I would want to test my Oozie Workflow. But I share your concerns : Is it really necessary to test both Spark local and integration with Oozie ... I'm not really theorical, so I need to get my hands dirty to really understand what's going on and what is useful. For now, I'm not sure we're gonna use this Oozie testing all the time, may be just for some specifics use cases and/or tricky workflows)
Thanks for the adresses, I thought some variables would have been initialized by the framework to get theses independently from my installation. In my case, it doesn't really matter because all my configurations are by default.
Ok thanks for your reply.
Can I close the issue?
Feel free to post PR if docs or other stuff are wrong :P
Ok, thanks.
I close the issue, don't worry 😄
Hello Jetoile,
First of all, I'ld like to thank you for all this great work ! :)
I'm currently trying to improve our practices on my project and I try to introduce Oozie integration tests.
But for now I'm stuck in a "classNotfoundError".
For more details :
pom.xml
Test class :
Stack trace :
I dont know if you need more details, but if so, don't hesitate !