fingltd / 4mc

4mc - splittable lz4 and zstd in hadoop/spark/flink
Other
108 stars 36 forks source link

What is the idea behind design of footer? #5

Closed tesseract2048 closed 9 years ago

tesseract2048 commented 9 years ago

I've noticed that, to make stream splittable, we have to read footer section to determine block indexes. But in some context, it is hard to retrieve the file length, thus cannot easily seek to a relative position of the end of file. On the other hand, in these contexts, another field in HEADER which indicates the footer offset will be handy.

Can this be solved?

carlomedas commented 9 years ago

4mc was born for hadoop and similar distributed filesystems, where we have 2 main constraints: