search

twitter / hadoop-lzo

Refactored version of code.google.com/hadoop-gpl-compression for hadoop 0.20
GNU General Public License v3.0
546 stars 329 forks source link

No output when using index file #125

Closed sjgiorgi closed 7 years ago

sjgiorgi commented 7 years ago

I'm creating a index file via

`hadoop jar hadoop-lzo-0.4.21.jar com.hadoop.compression.lzo.DistributedLzoIndexer  myInput.lzo`

This successfully runs and I get myInput.lzo.index out. To test this I wrote a trivial map reduce program that reads in a file and then writes it out. When I set the myInput.lzo.index as the input I get an empty output. When I use the original file myInput.lzo everything seems to work and the decompressed version of the file is written to dfs.

sjlee commented 7 years ago

myInput.lzo.index is not an input file itself. An lzo index file itself exists to help the lzo input to handle the actual lzo file more efficiently.

sjgiorgi commented 7 years ago

Sorry, thanks for the clarification. A little confusing with the documentation saying "Running MR Jobs over Indexed Files: Now run any job, say wordcount, over the new file".

sjgiorgi commented 7 years ago

So how does one pass the index file? I run the above, get the .index file and then pass the original file through existing code and get a single split.

sjlee commented 7 years ago

You don't directly pass the index file. The path of the index file is implicitly assumed to be (lzo file name).index (see https://github.com/twitter/hadoop-lzo/blob/master/src/main/java/com/hadoop/compression/lzo/LzoIndex.java#L41). The lzo file should be large enough to cause multiple splits.

sjgiorgi commented 7 years ago

If I decompress the file and run it I get multiple splits, keeping it compressed I always get 1 split. This is happening across hundreds of files. Does this mean I'm not creating the index file properly?

sjlee commented 7 years ago

As long as you use the right input format (e.g. LzoTextInputFormat) and the file is big enough, you should see multiple splits. You might want to play with LzoIndex to see if you can find out more of your file.