search

azkaban / azkaban-plugins

Plugins for Azkaban.
https://azkaban.github.io
Apache License 2.0
130 stars 178 forks source link

Got error when executing Hive on Snappy supported Hadoop #252

Open changzong opened 7 years ago

changzong commented 7 years ago

Hi, I just tried to run hive script with azkaban 3.0 with the hive plugin, and this error came up:

"Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec."

My Hadoop supports SnappyCodec and every configuration in hadoop and azkaban plugin seem correct. I think the problem may come from missing native library configuration but I can't find any place to set. Any suggestion?

Thanks

changzong commented 7 years ago

I just knew how this came up. The problem is from my tez configuration. I set tez.runtime.compress to true to compress my intermediate data and use SnappyCodec as the compression method. After I set the config to false, every thing succeeded. I still don't understand why this parameter cannot be used.