search

qubole / rubix

Cache File System optimized for columnar formats and object stores
Apache License 2.0
183 stars 74 forks source link

Add support for NativeAzureFileSystem in Rubix #88

Closed abhishekdas99 closed 5 years ago

vrajat commented 6 years ago

I know that some work has been done in #91 to add AzureFileSystem support. Can you help me understand the issues with hadoop versions and azure filesystem versions in open source presto/spark and qubole presto/spark ?

abhishekdas99 commented 6 years ago

@vrajat NativeAzureFileSystem is added in open source hadoop in 2.7 branch. So if any wants to use Rubix with Azure Blob, they need to have either hadoop 2.7 or higher version or they need to port NativeAzureFileSystem related code in their hadoop. In Qubole, we have ported NativeAzureFileSystem related code in our hadoop 2.6 version.

To conclude, we are adding version dependency of 2.7 of hadoop-azure module for compilation. NativeAzureFileSystem is not packaged in the rubix-core jar. In runtime, you need this NativeAzureFileSystem class in your classpath to get this to work.