search

trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
https://trino.io
Apache License 2.0
10.55k stars 3.03k forks source link

ADLS Gen2 Storage account not readable #18280

Closed pulkitkalia1994 closed 3 months ago

pulkitkalia1994 commented 1 year ago

Hi!

I am using latest version of Trino on Kubernetes and getting the below error from hive-

Please note that hive is able to connect and verify the path but unable to proceed.

Error stack -

23/07/13 17:45:02 [pool-5-thread-13]: DEBUG azurebfs.AzureBlobFileSystem: Initializing AzureBlobFileSystem for abfs://trinotestcontainer@trinoteststore.dfs.core.windows.net/year=2023/month=5/day=10
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG store.DataBlocks: Creating DataFactory of type : disk
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsPerfTracker: AbfsPerfTracker configuration: false
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Initializing SSL Context to channel mode Default
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Attempting to register OpenSSL provider
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Failed to load OpenSSL. Falling back to the JSSE default.
java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: openssl.TLS, provider: openssl, class: org.wildfly.openssl.OpenSSLContextSPI$OpenSSLTLSContextSpi)
        at java.security.Provider$Service.newInstance(Provider.java:1617)
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
        at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
        at org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.bindToOpenSSLProvider(DelegatingSSLSocketFactory.java:206)
        at org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.initializeSSLContext(DelegatingSSLSocketFactory.java:159)
        at org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.<init>(DelegatingSSLSocketFactory.java:134)
        at org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.initializeDefaultFactory(DelegatingSSLSocketFactory.java:105)
        at org.apache.hadoop.fs.azurebfs.services.AbfsClient.<init>(AbfsClient.java:132)
        at org.apache.hadoop.fs.azurebfs.services.AbfsClient.<init>(AbfsClient.java:165)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1579)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:236)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:183)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2795)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2829)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2811)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
        at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:104)
        at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:140)
        at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:146)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1386)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1449)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(HiveMetaStore.java:1438)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
        at com.sun.proxy.$Proxy8.create_table(Unknown Source)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table.getResult(ThriftHiveMetastore.java:9170)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table.getResult(ThriftHiveMetastore.java:9154)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:103)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: WFOPENSSL0001 Could not find libssl, please make sure -Dorg.wildfly.openssl.path is set correctly. Attempted to find it at [/libssl.1.1.so, /liblibssl-1_1-x64.so, /libssl.so, /libssleay32.so, /liblibssl32.so, /liblibssl-1_1.so, /usr/hdp/2.6.3.0-235/hadoop/lib/native/Linux-amd64-64/libssl.1.1.so, /usr/hdp/2.6.3.0-235/hadoop/lib/native/Linux-amd64-64/liblibssl-1_1-x64.so, /usr/hdp/2.6.3.0-235/hadoop/lib/native/Linux-amd64-64/libssl.so, /usr/hdp/2.6.3.0-235/hadoop/lib/native/Linux-amd64-64/libssleay32.so, /usr/hdp/2.6.3.0-235/hadoop/lib/native/Linux-amd64-64/liblibssl32.so, /usr/hdp/2.6.3.0-235/hadoop/lib/native/Linux-amd64-64/liblibssl-1_1.so, /usr/hdp/2.6.3.0-235/hadoop/lib/native/libssl.1.1.so, /usr/hdp/2.6.3.0-235/hadoop/lib/native/liblibssl-1_1-x64.so, /usr/hdp/2.6.3.0-235/hadoop/lib/native/libssl.so, /usr/hdp/2.6.3.0-235/hadoop/lib/native/libssleay32.so, /usr/hdp/2.6.3.0-235/hadoop/lib/native/liblibssl32.so, /usr/hdp/2.6.3.0-235/hadoop/lib/native/liblibssl-1_1.so]
        at org.wildfly.openssl.SSL.init(SSL.java:189)
        at org.wildfly.openssl.OpenSSLContextSPI.<init>(OpenSSLContextSPI.java:129)
        at org.wildfly.openssl.OpenSSLContextSPI$OpenSSLTLSContextSpi.<init>(OpenSSLContextSPI.java:463)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.security.Provider$Service.newInstance(Provider.java:1595)
        ... 38 more
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_RSA_WITH_AES_256_GCM_SHA384 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_RSA_WITH_AES_128_GCM_SHA256 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_DH_anon_WITH_AES_256_GCM_SHA384 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG ssl.DelegatingSSLSocketFactory: Removed Cipher - TLS_DH_anon_WITH_AES_128_GCM_SHA256 from list of enabled SSLSocket ciphers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsClientThrottlingIntercept: Client-side throttling is enabled for the ABFS file system.
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG azurebfs.AzureBlobFileSystem: Initializing AzureBlobFileSystem for abfs://trinotestcontainer@trinoteststore.dfs.core.windows.net/year=2023/month=5/day=10 complete
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG azurebfs.AzureBlobFileSystem: AzureBlobFileSystem.getFileStatus path: abfs://trinotestcontainer@trinoteststore.dfs.core.windows.net/year=2023/month=5/day=10
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG impl.IOStatisticsStoreImpl: Incrementing counter op_get_file_status by 1 with final value 1
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG azurebfs.AzureBlobFileSystemStore: isNamespaceEnabled is UNKNOWN; fall back and determine through getAcl server call
Cannot convert Trilean.UNKNOWN to boolean
        at org.apache.hadoop.fs.azurebfs.enums.Trilean.toBoolean(Trilean.java:74)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:348)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:932)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:609)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:599)
        at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:469)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1392)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1449)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(HiveMetaStore.java:1438)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
        at com.sun.proxy.$Proxy8.create_table(Unknown Source)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table.getResult(ThriftHiveMetastore.java:9170)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table.getResult(ThriftHiveMetastore.java:9154)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:103)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG azurebfs.AzureBlobFileSystemStore: Get root ACL status
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG impl.IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 1
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsClient: First execution of REST operation - GetAcl
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG impl.IOStatisticsStoreImpl: Incrementing counter connections_made by 1 with final value 1
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsClient: Signing request with shared key
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.SharedKeyCredentials: Signing request with timestamp of Thu, 13 Jul 2023 17:45:02 GMT and signature SharedKey trinoteststore:ttO3paTeMsD6oHRN3zwwM66gna1mieR5qq4/jzUiKrA=
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils: Request Headers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   Accept-Charset=utf-8
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-version=2019-12-12
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   Accept=application/json, application/octet-stream
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-date=Thu, 13 Jul 2023 17:45:02 GMT
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   User-Agent=APN/1.0 Azure Blob FS/2.7.3.2.6.3.0-235 (OracleCorporation JavaJRE 1.8.0_191; Linux 5.15.0-1040-azure/amd64; SunJSSE-1.8; UNKNOWN/UNKNOWN) unknown
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-client-request-id=:f69cd7a9-e67f-4961-9358-f2f70d908de2:d5c9499d-f969-4312-94f1-ff347ca462e0:::GF:0
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   Content-Type=
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils: Response Headers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   HTTP Response=HTTP/1.1 200 OK
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-group=$superuser
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-version=2019-12-12
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   Server=Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   ETag="0x8DB823895E9C90C"
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-acl=user::rwx,group::r-x,other::---,default:user::rwx,default:group::r-x,default:other::---
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-permissions=rwxr-x---+
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   Last-Modified=Tue, 11 Jul 2023 17:59:34 GMT
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-request-id=f4efa348-b01f-0039-18b1-b5619d000000
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   Date=Thu, 13 Jul 2023 17:45:02 GMT
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-client-request-id=:f69cd7a9-e67f-4961-9358-f2f70d908de2:d5c9499d-f969-4312-94f1-ff347ca462e0:::GF:0
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-owner=$superuser
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG impl.IOStatisticsStoreImpl: Incrementing counter get_responses by 1 with final value 1
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG impl.IOStatisticsStoreImpl: Incrementing counter bytes_received by 0 with final value 0
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsClient: HttpRequest: GetAcl: 200,,,cid=:f69cd7a9-e67f-4961-9358-f2f70d908de2:d5c9499d-f969-4312-94f1-ff347ca462e0:::GF:0,rid=f4efa348-b01f-0039-18b1-b5619d000000,sent=0,recv=0,HEAD,https://trinoteststore.dfs.core.windows.net/trinotestcontainer/?upn=false&action=getAccessControl&timeout=90
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG azurebfs.AzureBlobFileSystemStore: getFileStatus filesystem: trinotestcontainer path: abfs://trinotestcontainer@trinoteststore.dfs.core.windows.net/year=2023/month=5/day=10 isNamespaceEnabled: true
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG impl.IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 2
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsClient: First execution of REST operation - GetPathStatus
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG impl.IOStatisticsStoreImpl: Incrementing counter connections_made by 1 with final value 2
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsClient: Signing request with shared key
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.SharedKeyCredentials: Signing request with timestamp of Thu, 13 Jul 2023 17:45:02 GMT and signature SharedKey trinoteststore:Ec3Q/x2O4ENqDT1RLhjJzhYQzAcWx7EBW5PB4KoB+Ec=
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils: Request Headers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   Accept-Charset=utf-8
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-version=2019-12-12
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   Accept=application/json, application/octet-stream
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-date=Thu, 13 Jul 2023 17:45:02 GMT
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   User-Agent=APN/1.0 Azure Blob FS/2.7.3.2.6.3.0-235 (OracleCorporation JavaJRE 1.8.0_191; Linux 5.15.0-1040-azure/amd64; SunJSSE-1.8; UNKNOWN/UNKNOWN) unknown
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-client-request-id=:f7622515-ea13-438e-88c1-8b5fb6b8ba96:d5c9499d-f969-4312-94f1-ff347ca462e0:::GF:0
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   Content-Type=
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils: Response Headers
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   HTTP Response=HTTP/1.1 200 OK
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-group=$superuser
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-version=2019-12-12
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   Server=Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   Last-Modified=Tue, 11 Jul 2023 18:24:35 GMT
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   Date=Thu, 13 Jul 2023 17:45:02 GMT
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-resource-type=directory
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-server-encrypted=true
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   ETag="0x8DB823C14545953"
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-permissions=rwxr-x---+
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   Content-Length=0
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-request-id=f4efa34c-b01f-0039-1bb1-b5619d000000
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-client-request-id=:f7622515-ea13-438e-88c1-8b5fb6b8ba96:d5c9499d-f969-4312-94f1-ff347ca462e0:::GF:0
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsIoUtils:   x-ms-owner=$superuser
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG impl.IOStatisticsStoreImpl: Incrementing counter get_responses by 1 with final value 2
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG impl.IOStatisticsStoreImpl: Incrementing counter bytes_received by 0 with final value 0
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG services.AbfsClient: HttpRequest: GetPathStatus: 200,,,cid=:f7622515-ea13-438e-88c1-8b5fb6b8ba96:d5c9499d-f969-4312-94f1-ff347ca462e0:::GF:0,rid=f4efa34c-b01f-0039-1bb1-b5619d000000,sent=0,recv=0,HEAD,https://trinoteststore.dfs.core.windows.net/trinotestcontainer/year%3D2023/month%3D5/day%3D10?upn=false&action=getStatus&timeout=90
23/07/13 17:45:02 [pool-5-thread-13]: DEBUG metastore.ObjectStore: Rollback transaction, isActive: true at:
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
23/07/13 17:45:02 [pool-5-thread-13]: ERROR metastore.RetryingHMSHandler: java.lang.NoSuchMethodError: org.apache.hadoop.fs.FileStatus.<init>(JZIJJJLorg/apache/hadoop/fs/permission/FsPermission;Ljava/lang/String;Ljava/lang/String;Lorg/apache/hadoop/fs/Path;Lorg/apache/hadoop/fs/Path;ZZZ)V
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore$VersionedFileStatus.<init>(AzureBlobFileSystemStore.java:1739)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore$VersionedFileStatus.<init>(AzureBlobFileSystemStore.java:1719)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:993)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:609)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:599)
        at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:469)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1392)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1449)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(HiveMetaStore.java:1438)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
        at com.sun.proxy.$Proxy8.create_table(Unknown Source)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table.getResult(ThriftHiveMetastore.java:9170)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table.getResult(ThriftHiveMetastore.java:9154)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:103)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Exception in thread "pool-5-thread-13" java.lang.NoSuchMethodError: org.apache.hadoop.fs.FileStatus.<init>(JZIJJJLorg/apache/hadoop/fs/permission/FsPermission;Ljava/lang/String;Ljava/lang/String;Lorg/apache/hadoop/fs/Path;Lorg/apache/hadoop/fs/Path;ZZZ)V
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore$VersionedFileStatus.<init>(AzureBlobFileSystemStore.java:1739)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore$VersionedFileStatus.<init>(AzureBlobFileSystemStore.java:1719)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:993)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:609)
        at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:599)
        at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:469)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1392)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1449)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(HiveMetaStore.java:1438)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
        at com.sun.proxy.$Proxy8.create_table(Unknown Source)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table.getResult(ThriftHiveMetastore.java:9170)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table.getResult(ThriftHiveMetastore.java:9154)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:103)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
23/07/13 17:45:12 [abfs-timer-client-throttling-analyzer-read]: DEBUG services.AbfsClientThrottlingAnalyzer:  read,          0,          0,          0,          0,   0.00, 10001,     0,     0
23/07/13 17:45:12 [abfs-timer-client-throttling-analyzer-write]: DEBUG services.AbfsClientThrottlingAnalyzer: write,          0,          0,          0,          0,   0.00, 10001,     0,     0
23/07/13 17:45:22 [abfs-timer-client-throttling-analyzer-read]: DEBUG services.AbfsClientThrottlingAnalyzer:  read,          0,          0,          0,          0,   0.00, 10000,     0,     0
23/07/13 17:45:22 [abfs-timer-client-throttling-analyzer-write]: DEBUG services.AbfsClientThrottlingAnalyzer: write,          0,          0,          0,          0,   0.00, 10000,     0,     0
23/07/13 17:45:32 [abfs-timer-client-throttling-analyzer-write]: DEBUG services.AbfsClientThrottlingAnalyzer: write,          0,          0,          0,          0,   0.00, 10009,     0,     0
23/07/13 17:45:32 [abfs-timer-client-throttling-analyzer-read]: DEBUG services.AbfsClientThrottlingAnalyzer:  read,          0,          0,          0,          0,   0.00, 10005,     0,     0
23/07/13 17:45:42 [abfs-timer-client-throttling-analyzer-read]: DEBUG services.AbfsClientThrottlingAnalyzer:  read,          0,          0,          0,          0,   0.00, 10000,     0,     0
23/07/13 17:45:42 [abfs-timer-client-throttling-analyzer-write]: DEBUG services.AbfsClientThrottlingAnalyzer: write,          0,          0,          0,          0,   0.00, 10002,     0,     0
23/07/13 17:45:52 [abfs-timer-client-throttling-analyzer-read]: DEBUG services.AbfsClientThrottlingAnalyzer:  read,          0,          0,          0,          0,   0.00, 10001,     0,     0
pulkitkalia1994 commented 1 year ago

I am using Hive 1.2.2 with additional azure JARS - https://repo1.maven.org/maven2/com/microsoft/azure/azure-storage/8.6.6/azure-storage-8.6.6.jar https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-azure/3.3.2/hadoop-azure-3.3.2.jar https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/3.3.2/hadoop-common-3.3.2.jar https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-azure-datalake/3.3.2/hadoop-azure-datalake-3.3.2.jar

electrum commented 1 year ago

My guess is that Hive 1.2.2 is too old to be compatible with Hadoop 3. You'll probably need to try an older Hadoop version like Hadoop 2.x, or upgrade to a newer Hive version.