search

prestodb / presto

The official home of the Presto distributed SQL query engine for big data
http://prestodb.io
Apache License 2.0
15.75k stars 5.29k forks source link

No support for ARM based architectures (Missing hadoop native lib) #17200

Open PostProd opened 2 years ago

PostProd commented 2 years ago

Currently, there prestodb doesn't support ARM architectures for two reasons. The first is the PrestoSystemRequirements class which doesn't include "aarch64" in the allowed os.arch values. The second is that the underlying dependency for our hadoop jars is from 'com.facebook.presto.hadoop' which doesn't currently include nativelibs/ for the aarch64 ARM types. This leaves two options:

Option 1: Have Facebook update their hadoop libraries to also build nativelibs/ for "aarch64" Option 2: Change the dependency on hadoop packaging to be some other package that does bundle nativelibs/ for "aarch64"

rohanpednekar commented 2 years ago

Hi @v-jizhang here is another user request on the same topic we discussed last week. Do you think you can take it as a priority? Thanks!

rohanpednekar commented 2 years ago

Discussed this with @imjalpreet,

the Hadoop dependency being used in Presto is a custom shaded version of Apache Hadoop. It is also open source and can be worked on updating to Hadoop 3 as well as adding the required Hadoop native libraries for arm. Custom shaded version is done to include only the required dependencies so that the size is not too big. So, Option 2 is not possible. The only way is to modify the current dependency to make it compatible for ARM.

v-jizhang commented 2 years ago
PostProd commented 2 years ago

Why is Hadoop 3 a requirement for native ARM libraries? Is it not feasible to recompile the native libs on Hadoop 2.x? Additionally, if we were to move to Hadoop 3, why not consider Hadoop 3.3.x since it upgrades the protobuf dependency from 2.5.0 to 3.7.1? This adds significant value since protobuf 2.5.0 is unsupported/deprecated and doesn't even build Hadoop libs out of box (requires headers for atomics).

Hadoop 3.3.0 Release Notes: https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/release/3.3.0/RELEASENOTES.3.3.0.html

vbvasa commented 1 year ago

@v-jizhang - It would be great if we can take this on priority. We would like to move our Presto On Spark clusters to AWS Graviton and are waiting for presto to support ARM64. Thanks in advance!

Akanksha-kedia commented 2 months ago

i m still facing issue chip : Apple M1 Pro but issue : java.lang.ExceptionInInitializerError at io.prestosql.plugin.hive.HdfsEnvironment$$FastClassByGuice$$2c8553d4.newInstance() at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32) at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112) at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120) at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32) at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112) at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120) at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213) at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111) at com.google.inject.Guice.createInjector(Guice.java:87) at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:276) at io.prestosql.plugin.hive.InternalHiveConnectorFactory.createConnector(InternalHiveConnectorFactory.java:117) at io.prestosql.plugin.hive.InternalHiveConnectorFactory.createConnector(InternalHiveConnectorFactory.java:77) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at io.prestosql.plugin.hive.HiveConnectorFactory.create(HiveConnectorFactory.java:63) at io.prestosql.connector.ConnectorManager.createConnector(ConnectorManager.java:353) at io.prestosql.connector.ConnectorManager.createCatalog(ConnectorManager.java:212) at io.prestosql.connector.ConnectorManager.createCatalog(ConnectorManager.java:204) at io.prestosql.connector.ConnectorManager.createCatalog(ConnectorManager.java:190) at io.prestosql.metadata.StaticCatalogStore.loadCatalog(StaticCatalogStore.java:88) at io.prestosql.metadata.StaticCatalogStore.loadCatalogs(StaticCatalogStore.java:68) at io.prestosql.server.Server.doStart(Server.java:119) at io.prestosql.server.Server.lambda$start$0(Server.java:73) at io.prestosql.$gen.Presto_unknown____20240504_042839_1.run(Unknown Source) at io.prestosql.server.Server.start(Server.java:73) at io.prestosql.server.PrestoServer.main(PrestoServer.java:38) Caused by: java.lang.RuntimeException: failed to load Hadoop native library at io.prestosql.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:59) at io.prestosql.plugin.hive.HdfsEnvironment.(HdfsEnvironment.java:39) ... 52 more Caused by: java.lang.RuntimeException: library not found: /nativelib/Mac_OS_X-aarch64/libhadoop.dylib at io.prestosql.hadoop.HadoopNative.loadLibrary(HadoopNative.java:89) at io.prestosql.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:47) ... 53 more

Screenshot 2024-05-04 at 10 01 10 AM also i tried exporting export DYLD_LIBRARY_PATH=/path/

Downloaded the library from the provided link and saved it to a suitable directory, say /Users/ak/lib.

Set the DYLD_LIBRARY_PATH environment variable to include the directory where saved the libhadoop.dylib file:

export DYLD_LIBRARY_PATH=/Users/ak/lib