Error: in atguigu-presto: readDirect unsupported in RemoteBlockReader

[Rock-Lee-520]

Hi, everyone I got an error which used presto to query deltalake's data , but it notices me this error.

I changed the configuration in presto/etc/catalog/hdfs.xml ,but it does not work.

`

dfs.client.use.legacy.blockreader

<value>false</value>

`

So, can anyone help me ?

[agrawalreetika]

@Rock-liyi Could you please share the full stack trace? Not sure if you have a similar issue described in this thread? - https://stackoverflow.com/questions/49613451/unable-to-connect-to-hdfs-data-node-from-remote-client

[Rock-Lee-520]

@Rock-liyi Could you please share the full stack trace? Not sure if you have a similar issue described in this thread? - https://stackoverflow.com/questions/49613451/unable-to-connect-to-hdfs-data-node-from-remote-client

Thank for your help , this is full stack trace. ` java.lang.UnsupportedOperationException: readDirect unsupported in RemoteBlockReader at org.apache.hadoop.hdfs.RemoteBlockReader.read(RemoteBlockReader.java:492) at org.apache.hadoop.hdfs.DFSInputStream$ByteBufferStrategy.doRead(DFSInputStream.java:789) at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:823) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:883) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:938) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:143) at org.apache.parquet.hadoop.util.H2SeekableInputStream$H2Reader.read(H2SeekableInputStream.java:81) at org.apache.parquet.hadoop.util.H2SeekableInputStream.readFully(H2SeekableInputStream.java:90) at org.apache.parquet.hadoop.util.H2SeekableInputStream.readFully(H2SeekableInputStream.java:75) at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:575) at org.apache.parquet.hadoop.ParquetFileReader.(ParquetFileReader.java:776) at org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:657) at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:152) at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) at com.github.mjakubowski84.parquet4s.ParquetIterableImpl$$anon$2.hasNext(ParquetReader.scala:113) at io.delta.standalone.internal.actions.CustomParquetIterator.hasNext(MemoryOptimizedLogReplay.scala:134) at io.delta.standalone.internal.actions.MemoryOptimizedLogReplay$$anon$1.$anonfun$ensureNextIterIsReady$3(MemoryOptimizedLogReplay.scala:83) at io.delta.standalone.internal.actions.MemoryOptimizedLogReplay$$anon$1.$anonfun$ensureNextIterIsReady$3$adapted(MemoryOptimizedLogReplay.scala:83) at scala.Option.exists(Option.scala:376) at io.delta.standalone.internal.actions.MemoryOptimizedLogReplay$$anon$1.ensureNextIterIsReady(MemoryOptimizedLogReplay.scala:83) at io.delta.standalone.internal.actions.MemoryOptimizedLogReplay$$anon$1.hasNext(MemoryOptimizedLogReplay.scala:92) at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:43) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at io.delta.standalone.internal.SnapshotImpl.loadTableProtocolAndMetadata(SnapshotImpl.scala:135) at io.delta.standalone.internal.SnapshotImpl.x$1$lzycompute(SnapshotImpl.scala:125) at io.delta.standalone.internal.SnapshotImpl.x$1(SnapshotImpl.scala:125) at io.delta.standalone.internal.SnapshotImpl.protocolScala$lzycompute(SnapshotImpl.scala:125) at io.delta.standalone.internal.SnapshotImpl.protocolScala(SnapshotImpl.scala:125) at io.delta.standalone.internal.SnapshotImpl.(SnapshotImpl.scala:243) at io.delta.standalone.internal.SnapshotManagement.createSnapshot(SnapshotManagement.scala:257) at io.delta.standalone.internal.SnapshotManagement.getSnapshotAtInit(SnapshotManagement.scala:224) at io.delta.standalone.internal.SnapshotManagement.$init$(SnapshotManagement.scala:37) at io.delta.standalone.internal.DeltaLogImpl.(DeltaLogImpl.scala:46) at io.delta.standalone.internal.DeltaLogImpl$.apply(DeltaLogImpl.scala:225) at io.delta.standalone.internal.DeltaLogImpl$.forTable(DeltaLogImpl.scala:207) at io.delta.standalone.internal.DeltaLogImpl.forTable(DeltaLogImpl.scala) at io.delta.standalone.DeltaLog.forTable(DeltaLog.java:136) at com.facebook.presto.delta.DeltaClient.loadDeltaTableLog(DeltaClient.java:151) at com.facebook.presto.delta.DeltaClient.getTable(DeltaClient.java:79) at com.facebook.presto.delta.DeltaMetadata.getTableHandle(DeltaMetadata.java:137) at com.facebook.presto.delta.DeltaMetadata.getTableHandle(DeltaMetadata.java:57) at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorMetadata.getTableHandle(ClassLoaderSafeConnectorMetadata.java:220) at com.facebook.presto.metadata.MetadataManager.getTableHandle(MetadataManager.java:331) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.lambda$visitTable$19(StatementAnalyzer.java:1258) at com.facebook.presto.common.RuntimeStats.profileNanos(RuntimeStats.java:135) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:1258) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:320) at com.facebook.presto.sql.tree.Table.accept(Table.java:53) at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:334) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.analyzeFrom(StatementAnalyzer.java:2509) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:1506) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:320) at com.facebook.presto.sql.tree.QuerySpecification.accept(QuerySpecification.java:138) at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:334) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:342) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:1088) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:320) at com.facebook.presto.sql.tree.Query.accept(Query.java:105) at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:334) at com.facebook.presto.sql.analyzer.StatementAnalyzer.analyze(StatementAnalyzer.java:312) at com.facebook.presto.sql.analyzer.Analyzer.analyzeSemantic(Analyzer.java:93) at com.facebook.presto.execution.SqlQueryExecution.(SqlQueryExecution.java:196) at com.facebook.presto.execution.SqlQueryExecution.(SqlQueryExecution.java:100) at com.facebook.presto.execution.SqlQueryExecution$SqlQueryExecutionFactory.createQueryExecution(SqlQueryExecution.java:814) at com.facebook.presto.dispatcher.LocalDispatchQueryFactory.lambda$createDispatchQuery$0(LocalDispatchQueryFactory.java:127) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

`

[agrawalreetika]

Hi @Rock-liyi, As per stacktrace it looks like some issue reading data from data nodes. Did you try propety given in this link?

<property>
  <name>dfs.client.use.datanode.hostname</name>
  <value>true</value>
  <description>Whether clients should use datanode hostnames when
    connecting to datanodes.
  </description>
</property>

[Rock-Lee-520]

Hi @Rock-liyi, As per stacktrace it looks like some issue reading data from data nodes. Did you try propety given in this link?

<property>
  <name>dfs.client.use.datanode.hostname</name>
  <value>true</value>
  <description>Whether clients should use datanode hostnames when
    connecting to datanodes.
  </description>
</property>

I tried to use this link, but it did not work.

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>master:50090</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>3</value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
        <value>file:/mnt/hadoop/tmp/name</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
        <value>file:/mnt/hadoop/tmp/data</value>
    </property>
    <property>
        <name>dfs.client.use.datanode.hostname</name>
        <value>true</value>
        <description>Whether clients should use datanode hostnames when
            connecting to datanodes.
        </description>
    </property>
    <property>
        <name>dfs.client.use.legacy.blockreader</name>
        <value>false</value>
    </property>

</configuration>

[Rock-Lee-520]

@agrawalreetika Could you help me please?

It will get error when query a big data file

[agrawalreetika]

Are you using presto-delta connector? And which version of presto are you on?

[Rock-Lee-520]

@agrawalreetika

[ITRS1991]

It could be that there are some problems with hadoop-apache2-2.7.4-9.jar，you can try removing core-site.xml to circumvent the problem, since the default is false。

[neshkeev]

Has anyone managed to come up with a workaround? Can the delta connector be configured to read some external Hadoop configuration?

[agrawalreetika]

@daniellj Posted solution for above issue what worked for him here - https://github.com/prestodb/presto/issues/21283