search

fingltd / 4mc

4mc - splittable lz4 and zstd in hadoop/spark/flink
Other
108 stars 36 forks source link

Update FourMcNativeCodeLoader.java #6

Closed ankurbarua closed 9 years ago

ankurbarua commented 9 years ago

Allows loading the native libraries from LD_LIBRARY_PATH using Distributed Cache in Hadoop. Otherwise we need to put the native libraries in the $HADOOP_HOME/lib/native in all nodes.

carlomedas commented 9 years ago

Hello, please tell me more about this: as far as I understood you would like to setup an automatic failover, working only on Linux/Unix, in case library cannot be loaded at first try. The idea is not bad, although I'm surprised you do not want to put the libraries in the nodes, and you'll be forced to add them to each single job Distributed Cache.

ankurbarua commented 9 years ago

i would love to put them in the nodes as part of Hadoop distribution :-). This is great actually.

I think there should be a way for end users of hadoop who almost always don't have admin priv to install stuff into nodes.