search

twitter / hadoop-lzo

Refactored version of code.google.com/hadoop-gpl-compression for hadoop 0.20
GNU General Public License v3.0
546 stars 329 forks source link

Hadoop LZO does not take non-default queue #126

Closed rajitsaha closed 6 years ago

rajitsaha commented 7 years ago

We are using HDP 2.4 with hadoop-lzo-0.6.0.2.4.0.0-169.jar.

It seems hadoop-lzo does not take non-default queue name

I tried both of the following , hadoop jar /usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.4.0.0-169.jar com.hadoop.compression.lzo.DistributedLzoIndexer -Dmapred.job.queue.name=batch /user/rsaha/part-m-00000.lzo

Or

hadoop jar /usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.4.0.0-169.jar com.hadoop.compression.lzo.DistributedLzoIndexer -Dmapreduce.job.queuename=batch /user/rsaha/part-m-00000.lzo

in both the cases the MapReduce job is going to "default" queue

Whereas when I try other MapReduce Job with same option I can submit the job in non-default "batch" queue

hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar pi -Dmapred.job.queue.name=batch 10 10

jrottinghuis commented 6 years ago

With https://github.com/twitter/hadoop-lzo/pull/127 merged, can this issue be closed @rajitsaha?