Hello, I try to use 4mc codecs. I copied .so libraries and .jar file to my ../lib and /lib/native on all nodes. I also set core-site.xml on all nodes. See below. Other codecs residing in this paths work well unless fourmc. Is there something I forgot do do? Looks like Hadoop cannot find where classes are located? Running HDP 2.4.0.
I did not compile by myself, just downloaded .so and .jar file. Excuse my naivity, I am new to this.
Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.fourmc.Lz4Codec was not found.
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:122)
at org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.getRecordWriter(RawKeyTextOutputFormat.java:89)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.fourmc.Lz4Codec not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:119)
... 9 more
`
Hello, I try to use 4mc codecs. I copied .so libraries and .jar file to my ../lib and /lib/native on all nodes. I also set core-site.xml on all nodes. See below. Other codecs residing in this paths work well unless fourmc. Is there something I forgot do do? Looks like Hadoop cannot find where classes are located? Running HDP 2.4.0.
I did not compile by myself, just downloaded .so and .jar file. Excuse my naivity, I am new to this.
Any help appreciated. Many thanks!
cp 4mc-master/java/hadoop-4mc/src/main/resources/com/hadoop/compression/fourmc/linux/amd64/libhadoop-4mc.so /usr/hdp/2.4.0.0-169/hadoop/lib/nativecp 4mc/hadoop-4mc-2.0.0.jar /usr/hdp/2.4.0.0-169/hadoop/lib<name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,com.hadoop.compression.fourmc.Lz4Codec,com.hadoop.compression.fourmc.Lz4MediumCodec,com.hadoop.compression.fourmc.Lz4HighCodec,com.hadoop.compression.fourmc.Lz4UltraCodec,com.hadoop.compression.fourmc.FourMcCodec,com.hadoop.compression.fourmc.FourMcMediumCodec,com.hadoop.compression.fourmc.FourMcHighCodec,com.hadoop.compression.fourmc.FourMcUltraCodec,com.hadoop.compression.fourmc.FourMzCodec,com.hadoop.compression.fourmc.FourMzMediumCodec,com.hadoop.compression.fourmc.FourMzHighCodec,com.hadoop.compression.fourmc.FourMzUltraCodec</value> </property>When trying to use codec within sqoop tool I see following output
` 17/03/27 15:41:58 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 17/03/27 15:41:58 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 0d76fa3d79ee46cc5bc1f2bdad607fb18978826d] 17/03/27 15:41:58 INFO fourmc.FourMcNativeCodeLoader: hadoop-4mc: loaded native library (embedded) 17/03/27 15:41:58 INFO fourmc.Lz4Codec: Successfully loaded & initialized native-4mc library 17/03/27 15:41:58 INFO fourmc.Lz4MediumCodec: Successfully loaded & initialized native-4mc library 17/03/27 15:41:58 INFO fourmc.Lz4HighCodec: Successfully loaded & initialized native-4mc library 17/03/27 15:41:58 INFO fourmc.Lz4UltraCodec: Successfully loaded & initialized native-4mc library 17/03/27 15:41:58 INFO fourmc.ZstdCodec: Successfully loaded & initialized native-4mc library
Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.fourmc.Lz4Codec was not found. at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:122) at org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.getRecordWriter(RawKeyTextOutputFormat.java:89) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.fourmc.Lz4Codec not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:119)
... 9 more
`
Jan