stale[bot]
commented
2 years ago This issue has been automatically marked as stale because it has not had any activity in the last 2 years. If you feel that this issue is important, just comment and the stale tag will be removed; otherwise it will be closed in 7 days. This is an attempt to ensure that our open issues remain valuable and relevant so that we can keep track of what needs to be done and prioritize the right things.
Hello. I have Presto EMR 5.29.0 cluster.
I was able to create hive schema like this:
CREATE SCHEMA hive.bigdata WITH (location = 's3://vogo-big-data-lake/');and also able to create one table in it.But when I was creating another table which has lot of files in s3, I was getting this error in client side:
com.facebook.presto.spi.PrestoException: <private-ip>:9083: java.net.SocketTimeoutException: Read timed out at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.alterTable(ThriftHiveMetastore.java:890) at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.updateTableStatistics(ThriftHiveMetastore.java:407) at com.facebook.presto.hive.metastore.thrift.BridgingHiveMetastore.updateTableStatistics(BridgingHiveMetastore.java:112) at com.facebook.presto.hive.metastore.CachingHiveMetastore.updateTableStatistics(CachingHiveMetastore.java:338) at com.facebook.presto.hive.metastore.CachingHiveMetastore.updateTableStatistics(CachingHiveMetastore.java:338) at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore$UpdateStatisticsOperation.run(SemiTransactionalHiveMetastore.java:2560) at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore$Committer.executeUpdateStatisticsOperations(SemiTransactionalHiveMetastore.java:1423) at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore$Committer.access$1300(SemiTransactionalHiveMetastore.java:1064) at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore.commitShared(SemiTransactionalHiveMetastore.java:1006) at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore.commit(SemiTransactionalHiveMetastore.java:901) at com.facebook.presto.hive.HiveMetadata.commit(HiveMetadata.java:2518) at com.facebook.presto.hive.HiveConnector.commit(HiveConnector.java:211) at com.facebook.presto.transaction.InMemoryTransactionManager$TransactionMetadata$ConnectorTransactionMetadata.commit(InMemoryTransactionManager.java:601) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.thrift.transport.TTransportException: ip-172-31-36-35.ap-south-1.compute.internal:9083: java.net.SocketTimeoutException: Read timed out at com.facebook.presto.hive.metastore.thrift.Transport.rewriteException(Transport.java:91) at com.facebook.presto.hive.metastore.thrift.Transport.access$000(Transport.java:32) at com.facebook.presto.hive.metastore.thrift.Transport$TTransportWrapper.readAll(Transport.java:168) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_table(ThriftHiveMetastore.java:1316) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_table(ThriftHiveMetastore.java:1301) at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastoreClient.alterTable(ThriftHiveMetastoreClient.java:141) at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.lambda$alterTable$40(ThriftHiveMetastore.java:881) at com.facebook.presto.hive.metastore.thrift.HiveMetastoreApiStats.lambda$wrap$0(HiveMetastoreApiStats.java:42) at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:139) at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.alterTable(ThriftHiveMetastore.java:875) ... 20 more Suppressed: org.apache.thrift.transport.TTransportException: ip-172-31-36-35.ap-south-1.compute.internal:9083: java.net.SocketTimeoutException: Read timed out ... 34 more Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at com.facebook.presto.hive.metastore.thrift.Transport$TTransportWrapper.readAll(Transport.java:165) ... 31 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 33 more Suppressed: org.apache.thrift.transport.TTransportException: ip-172-31-36-35.ap-south-1.compute.internal:9083: java.net.SocketTimeoutException: Read timed out ... 34 more Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at com.facebook.presto.hive.metastore.thrift.Transport$TTransportWrapper.readAll(Transport.java:165) ... 31 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 33 more Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at com.facebook.presto.hive.metastore.thrift.Transport$TTransportWrapper.readAll(Transport.java:165) ... 31 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 33 moreAnd this in server side
ERROR transaction-finishing-0 com.google.common.util.concurrent.AbstractFuture RuntimeException while executing runnable CallbackListener{com.facebook.presto.execution.QueryStateMachine$1@e721eca} with executor MoreExecutors.directExecutor() com.facebook.presto.spi.PrestoException: Unknown transaction ID: 6e5b104f-5571-4f14-a1a0-5a3a7479ad6e. Possibly expired? Commands ignored until end of transaction block at com.facebook.presto.transaction.InMemoryTransactionManager.unknownTransactionError(InMemoryTransactionManager.java:280) at com.facebook.presto.transaction.InMemoryTransactionManager.getTransactionMetadata(InMemoryTransactionManager.java:259) at com.facebook.presto.transaction.InMemoryTransactionManager.getTransactionInfo(InMemoryTransactionManager.java:157) at com.facebook.presto.transaction.TransactionManager.isAutoCommit(TransactionManager.java:37) at com.facebook.presto.execution.QueryStateMachine.lambda$transitionToFailed$14(QueryStateMachine.java:817) at java.util.Optional.ifPresent(Optional.java:159) at com.facebook.presto.execution.QueryStateMachine.transitionToFailed(QueryStateMachine.java:816) at com.facebook.presto.execution.QueryStateMachine$1.onFailure(QueryStateMachine.java:784) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1341) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1027) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:868) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:754) at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:1020) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1027) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:868) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:754) at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:1020) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1027) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:868) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:713) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:93) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1027) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:868) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:713) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:133) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:80) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Following are the default properties set up by cluster: hive.properties:
hive.metastore-refresh-interval=1m connector.name=hive-hadoop2 hive.metastore-cache-ttl=20m hive.config.resources=/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml hive.non-managed-table-writes-enabled = true hive.s3-file-system-type = EMRFS hive.hdfs.impersonation.enabled = true hive.metastore.uri = thrift://:9083
config.properties:
coordinator=true node-scheduler.include-coordinator=false discovery.uri=http://:8889
http-server.threads.max=500
discovery-server.enabled=true
sink.max-buffer-size=1GB
query.max-memory=6452.0MB
query.max-memory-per-node=4256312590B
query.max-total-memory-per-node=5107575108B
query.max-history=40
query.min-expire-age=30m
query.client.timeout=30m
query.stage-count-warning-threshold=100
query.max-stage-count=150
http-server.http.port=8889
http-server.log.path=/var/log/presto/http-request.log
http-server.log.max-size=67108864B
http-server.log.max-history=5
log.max-size=268435456B
log.max-history=5