cannot compile with 2.6.0

alex-dot commented 9 years ago

This may be related to gradle/maven but I cannot seem to get this build for hadoop 2.6.0. Changing the versions in trace-mr2/build.gradle to 2.6.0 crashes the build process, stating

/srv/invisio/inviso/trace-mr2/src/main/java/com/netflix/bdp/inviso/fs/S3DelegateFS.java:29: error: package org.apache.hadoop.fs.s3native does not exist

Is there any other place I have to change the hadoop version? Or something else I am missing?

danielcweeks commented 9 years ago

Looks like the NativeS3FileSystem is packaged differently in 2.6.0. Just add compile 'org.apache.hadoop:hadoop-aws:2.6.0' as a dependency and it should work.

diff --git a/trace-mr2/build.gradle b/trace-mr2/build.gradle
index 583012f..ebd0f34 100644
--- a/trace-mr2/build.gradle
+++ b/trace-mr2/build.gradle
@@ -26,9 +26,10 @@ dependencies {
        compile 'org.apache.commons:commons-lang3:3.2.1'
     compile 'log4j:log4j:1.2.17'
     compile 'org.glassfish.jersey:project:2.4.1'
-    compile 'org.apache.hadoop:hadoop-common:2.4.0'
-    compile 'org.apache.hadoop:hadoop-client:2.4.0'
-    compile 'org.apache.hadoop:hadoop-mapreduce:2.4.0'
+    compile 'org.apache.hadoop:hadoop-common:2.6.0'
+    compile 'org.apache.hadoop:hadoop-aws:2.6.0'
+    compile 'org.apache.hadoop:hadoop-client:2.6.0'
+    compile 'org.apache.hadoop:hadoop-mapreduce:2.6.0'
     compile 'net.sf.ehcache:ehcache:2.7.4'
     compile 'xerces:xercesImpl:2.11.0'
     compile 'xalan:xalan:2.7.1'

alex-dot commented 9 years ago

Ah yes. Thought it might be something that simple, thanks for pointing to the correct package!

Netflix / inviso

cannot compile with 2.6.0 #8