search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.42k stars 953 forks source link

[Bug] Write paimon table with hive engine by MR got exeception #4537

Open FrommyMind opened 23 hours ago

FrommyMind commented 23 hours ago

Search before asking

Paimon version

0.9

Compute Engine

hive: 2.1-cdh-6.3-1

Minimal reproduce step

create paimon table in beeline

SET hive.metastore.warehouse.dir=/user/hive/warehouse;

CREATE TABLE hive_test_table(
    a INT COMMENT 'The a field',
    b STRING COMMENT 'The b field'
)
STORED BY 'org.apache.paimon.hive.PaimonStorageHandler';

after that, try insert data

insert into hive_test_table values (3, 'paimon');

What doesn't meet your expectations?

MapReduce failed with bellow error.

2024-11-17 22:11:46,830 INFO [main] org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[0]
2024-11-17 22:11:46,830 INFO [main] org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing operator SEL[1]
2024-11-17 22:11:47,196 INFO [main] org.apache.hadoop.hive.ql.exec.SelectOperator: SELECT struct<tmp_values_col1:string,tmp_values_col2:string>
2024-11-17 22:11:47,203 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: Initializing operator FS[3]
2024-11-17 22:11:47,204 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
2024-11-17 22:11:47,621 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: Using serializer : org.apache.paimon.hive.PaimonSerDe@6ede46f6 and formatter : org.apache.paimon.hive.mapred.PaimonOutputFormat@66273da0
2024-11-17 22:11:47,621 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.healthChecker.script.timeout is deprecated. Instead, use mapreduce.tasktracker.healthchecker.script.timeout
2024-11-17 22:11:47,634 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS hdfs://cdh01.daniel.com:8020/user/hive/warehouse/default.db/_tmp.hive_test_table/000000_0
2024-11-17 22:11:47,781 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[3]: records written - 1
2024-11-17 22:11:47,913 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.zstd]
2024-11-17 22:11:48,249 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: MAP[0]: records read - 1
2024-11-17 22:11:48,249 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: MAP[0]: Total records read - 1. abort - false
2024-11-17 22:11:48,249 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0, RECORDS_IN:1, 
2024-11-17 22:11:48,250 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[3]: records written - 1
2024-11-17 22:11:48,250 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: TOTAL_TABLE_ROWS_WRITTEN:1, RECORDS_OUT_1_default.hive_test_table:1, 
2024-11-17 22:11:48,272 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1731851152753_0005_m_000000_0 is done. And is in the process of committing
2024-11-17 22:11:48,276 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1731851152753_0005_m_000000_0 is allowed to commit now
2024-11-17 22:11:48,357 ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: com.github.luben.zstd.ZstdOutputStreamNoFinalizer.recommendedCOutSize()J
    at com.github.luben.zstd.ZstdOutputStreamNoFinalizer.recommendedCOutSize(Native Method)
    at com.github.luben.zstd.ZstdOutputStreamNoFinalizer.<clinit>(ZstdOutputStreamNoFinalizer.java:30)
    at com.github.luben.zstd.RecyclingBufferPool.<clinit>(RecyclingBufferPool.java:17)
    at org.apache.paimon.shade.org.apache.parquet.hadoop.codec.ZstandardCodec.createOutputStream(ZstandardCodec.java:107)
    at org.apache.paimon.shade.org.apache.parquet.hadoop.codec.ZstandardCodec.createOutputStream(ZstandardCodec.java:100)
    at org.apache.paimon.shade.org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.compress(CodecFactory.java:176)
    at org.apache.paimon.shade.org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:168)
    at org.apache.paimon.shade.org.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:59)
    at org.apache.paimon.shade.org.apache.parquet.column.impl.ColumnWriterBase.writePage(ColumnWriterBase.java:389)
    at org.apache.paimon.shade.org.apache.parquet.column.impl.ColumnWriteStoreBase.flush(ColumnWriteStoreBase.java:186)
    at org.apache.paimon.shade.org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:29)
    at org.apache.paimon.shade.org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:185)
    at org.apache.paimon.shade.org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:124)
    at org.apache.paimon.shade.org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:112)
    at org.apache.paimon.format.parquet.writer.ParquetBulkWriter.close(ParquetBulkWriter.java:52)
    at org.apache.paimon.io.SingleFileWriter.close(SingleFileWriter.java:170)
    at org.apache.paimon.io.RowDataFileWriter.close(RowDataFileWriter.java:104)
    at org.apache.paimon.io.RollingFileWriter.closeCurrentWriter(RollingFileWriter.java:131)
    at org.apache.paimon.io.RollingFileWriter.close(RollingFileWriter.java:168)
    at org.apache.paimon.append.AppendOnlyWriter$DirectSinkWriter.flush(AppendOnlyWriter.java:418)
    at org.apache.paimon.append.AppendOnlyWriter.flush(AppendOnlyWriter.java:219)
    at org.apache.paimon.append.AppendOnlyWriter.prepareCommit(AppendOnlyWriter.java:207)
    at org.apache.paimon.operation.AbstractFileStoreWrite.prepareCommit(AbstractFileStoreWrite.java:210)
    at org.apache.paimon.operation.MemoryFileStoreWrite.prepareCommit(MemoryFileStoreWrite.java:152)
    at org.apache.paimon.table.sink.TableWriteImpl.prepareCommit(TableWriteImpl.java:253)
    at org.apache.paimon.table.sink.TableWriteImpl.prepareCommit(TableWriteImpl.java:260)
    at org.apache.paimon.hive.mapred.PaimonOutputCommitter.commitTask(PaimonOutputCommitter.java:95)
    at org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:343)
    at org.apache.hadoop.mapred.Task.commit(Task.java:1341)
    at org.apache.hadoop.mapred.Task.done(Task.java:1185)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:351)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

2024-11-17 22:11:48,460 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
2024-11-17 22:11:48,460 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
2024-11-17 22:11:48,461 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.

Anything else?

I checked if my cluster supports zstd compression: Use the following command to run MR job

hadoop jar /opt/cloudera/parcels/CDH/jars/hadoop-mapreduce-examples*.jar wordcount -Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.ZStandardCodec -Dmapreduce.map.output.compress=true -Dmapreduce.output.fileoutputformat.compress=true -Dmapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.ZStandardCodec wcin wcout-zst

The job ran successfully.

a part of that log.

2024-11-17 22:07:43,023 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2024-11-17 22:07:43,410 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-11-17 22:07:43,410 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-11-17 22:07:43,421 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2024-11-17 22:07:43,565 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://cdh01.daniel.com:8020/user/hive/wcin/yum.log:0+0
2024-11-17 22:07:43,672 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 67108860(268435440)
2024-11-17 22:07:43,672 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 256
2024-11-17 22:07:43,672 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 214748368
2024-11-17 22:07:43,672 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 268435456
2024-11-17 22:07:43,672 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 67108860; length = 16777216
2024-11-17 22:07:43,679 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2024-11-17 22:07:43,696 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output
2024-11-17 22:07:43,715 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.zst]
2024-11-17 22:07:43,739 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1731851152753_0004_m_000000_0 is done. And is in the process of committing
2024-11-17 22:07:43,770 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1731851152753_0004_m_000000_0' done.
2024-11-17 22:07:43,778 INFO [main] org.apache.hadoop.mapred.Task: Final Counters for attempt_1731851152753_0004_m_000000_0: Counters: 29
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=220970
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=116
        HDFS: Number of bytes written=0
        HDFS: Number of read operations=3
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=0
        HDFS: Number of bytes read erasure-coded=0
    Map-Reduce Framework
        Map input records=0
        Map output records=0
        Map output bytes=0
        Map output materialized bytes=90
        Input split bytes=116
        Combine input records=0
        Combine output records=0
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=41
        CPU time spent (ms)=430
        Physical memory (bytes) snapshot=482156544
        Virtual memory (bytes) snapshot=2589884416
        Total committed heap usage (bytes)=480247808
        Peak Map Physical memory (bytes)=482156544
        Peak Map Virtual memory (bytes)=2589884416
    File Input Format Counters 
        Bytes Read=0
2024-11-17 22:07:43,879 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
2024-11-17 22:07:43,879 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
2024-11-17 22:07:43,879 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.

Are you willing to submit a PR?

ChrisKyrie commented 5 hours ago

I encountered the same problem and solved it by upgrading the zstd-jni version to zstd-jni-1.5.5-11 https://github.com/apache/paimon/issues/4531

FrommyMind commented 4 hours ago

I tied with below steps:

  1. put zstd-jni-1.5.5-11.jar under hive auxlib path.
  2. restart hiveserver2
  3. insert data to paimon table.

the problem still there.

the two version zstd-jni both in class.path.

java.class.path: /data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002:/etc/hadoop/conf.cloudera.yarn2:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/accessors-smart-1.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/netty-3.10.6.Final.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-mapreduce-client-uploader-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/woodstox-core-5.0.3.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jersey-servlet-1.19.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-yarn-server-common-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/aopalliance-1.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/slf4j-api-1.7.25.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/protobuf-java-2.5.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jersey-json-1.19.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-common.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/zookeeper.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/xz-1.6.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/logredactor-2.0.7.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-hadoop.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jul-to-slf4j-1.7.25.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/re2j-1.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jetty-util-9.3.25.v20180904.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/commons-configuration2-2.1.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jackson-core-2.9.9.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/json-simple-1.1.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-archives-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerby-asn1-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/commons-cli-1.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerb-crypto-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jetty-webapp-9.3.25.v20180904.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jackson-mapper-asl-1.9.13-cloudera.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-encoding.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-thrift.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-generator.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/netty-handler-4.1.17.Final.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/okhttp-2.7.5.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/audience-annotations-0.5.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-azure-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jetty-server-9.3.25.v20180904.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jettison-1.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/objenesis-1.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jackson-jaxrs-base-2.9.9.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/mssql-jdbc-6.2.1.jre7.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/commons-lang3-3.7.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/log4j-core-2.8.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/stax2-api-3.1.4.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/commons-math3-3.1.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/commons-compress-1.18.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jaxb-impl-2.2.3-1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-mapreduce-client-jobclient-3.0.0-cdh6.3.2-tests.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/commons-net-3.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jersey-core-1.19.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-pig-bundle.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/avro-1.8.2-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-hdfs-client-3.0.0-cdh6.3.2-tests.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-archive-logs-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/httpclient-4.5.3.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerb-server-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/fst-2.50.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerb-admin-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/event-publish-6.3.1-shaded.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerby-util-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/commons-codec-1.11.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-format-sources.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/javax.activation-api-1.2.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-sls-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerby-pkix-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jcip-annotations-1.0-1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/bcpkix-jdk15on-1.60.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/javax.servlet-api-3.1.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/json-io-2.5.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-yarn-api-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jackson-databind-2.9.9.3.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/lz4-java-1.5.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/slf4j-log4j12.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/commons-collections-3.2.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/gson-2.2.4.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-kms-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/zstd-jni-1.3.8-1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/azure-keyvault-core-0.8.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/commons-io-2.6.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jetty-util-ajax-9.3.25.v20180904.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-yarn-applications-distributedshell-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jetty-http-9.3.25.v20180904.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-resourceestimator-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jersey-guice-1.19.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/guice-servlet-4.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jsp-api-2.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jsr311-api-1.1.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-aliyun-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/commons-logging-1.1.3.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/asm-5.0.4.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jackson-xc-1.9.13.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-common-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerb-util-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jackson-core-asl-1.9.13.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-yarn-registry-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-annotations-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jsch-0.1.54.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-streaming-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-aws-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-hadoop-bundle.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-mapreduce-client-core-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-yarn-server-web-proxy-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jetty-io-9.3.25.v20180904.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/curator-recipes-2.12.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jaxb-api-2.2.11.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-datajoin-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jersey-server-1.19.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/htrace-core4-4.1.0-incubating.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jackson-annotations-2.9.9.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jetty-xml-9.3.25.v20180904.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/gcs-connector-hadoop3-1.9.10-cdh6.3.2-shaded.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/curator-framework-2.12.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/zookeeper-3.4.5-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-mapreduce-client-hs-plugins-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-rumen-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-cascading3.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-yarn-server-tests-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jdom-1.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/spark-2.4.0-cdh6.3.2-yarn-shuffle.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/avro.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/netty-transport-4.1.17.Final.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jsr305-3.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/okio-1.6.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-column.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-format-javadoc.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/metrics-core-3.0.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-yarn-client-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerby-config-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerb-core-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/ojalgo-43.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-hdfs-native-client-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/commons-daemon-1.0.13.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kafka-clients-2.2.1-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/tt-instrumentation-6.3.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-protobuf.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/guice-4.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/aws-java-sdk-bundle-1.11.271.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/azure-data-lake-store-sdk-2.2.9.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/bcprov-jdk15on-1.60.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jackson-jaxrs-1.9.13.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-hdfs-3.0.0-cdh6.3.2-tests.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/json-smart-2.3.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jersey-client-1.19.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-nfs-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/netty-codec-4.1.17.Final.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-mapreduce-client-common-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerby-xdr-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-hdfs-client-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/paranamer-2.8.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-hdfs-httpfs-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-mapreduce-client-nativetask-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-yarn-common-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/HikariCP-java7-2.4.12.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-auth-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/commons-lang-2.6.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerb-identity-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-mapreduce-client-hs-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-hdfs-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-pig.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerb-simplekdc-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-scala_2.11.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-distcp-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerb-client-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/netty-common-4.1.17.Final.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/azure-storage-5.4.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-jackson.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/java-util-1.9.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-yarn-server-nodemanager-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/javax.inject-1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/snappy-java-1.1.4.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-avro.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-kafka-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/ehcache-3.3.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/kerb-common-1.0.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-format.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/commons-beanutils-1.9.4.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/parquet-cascading.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jetty-servlet-9.3.25.v20180904.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/guava-11.0.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-openstack-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/aliyun-sdk-oss-2.8.3.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/nimbus-jose-jwt-4.41.1.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-mapreduce-client-app-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-gridmix-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/curator-client-2.12.0.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jackson-jaxrs-json-provider-2.9.9.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jetty-security-9.3.25.v20180904.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-hdfs-nfs-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/log4j-1.2.17.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-yarn-applications-unmanaged-am-launcher-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-mapreduce-client-jobclient-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/netty-codec-http-4.1.17.Final.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/jackson-module-jaxb-annotations-2.9.9.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-common-3.0.0-cdh6.3.2-tests.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-azure-datalake-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/netty-resolver-4.1.17.Final.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-mapreduce-client-shuffle-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-extras-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-hdfs-native-client-3.0.0-cdh6.3.2-tests.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/wildfly-openssl-1.0.4.Final.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/hadoop-mapreduce-examples-3.0.0-cdh6.3.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/netty-buffer-4.1.17.Final.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/httpcore-4.4.6.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/log4j-api-2.8.2.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/mr-framework/leveldbjni-all-1.8.jar::job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/job.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/hive-exec-2.1.1-cdh6.3.2-core.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/hive-exec-core.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/paimon-hive-connector-2.1-cdh-6.3-1.0-20241109.002448-47.jar:/data1/yarn/nm/usercache/hive/appcache/application_1726120453401_1470/container_1726120453401_1470_01_000002/zstd-jni-1.5.5-11.jar
ChrisKyrie commented 4 hours ago

Try the following steps:

  1. delete zstd-jni-1.3.8-1.jar under hive/lib
  2. put zstd-jni-1.5.5-11.jar under hive/lib
  3. restart hiveserver2
  4. insert data to paimon table.
FrommyMind commented 3 hours ago

@ChrisKyrie , Thank you qulickly answer.

my cluster is CHD6.3.2 , the zstd-jni not under hive/lib. it under mr/lib image

I don't think it's a good idea to remove the zstd-jni-1.3.8.jar on all yarn nodemanager nodes.

I'm not sure if this will cause other problems.

ChrisKyrie commented 2 hours ago

The essence of the problem is that zstd-jni-1.3.8-1.jar does not contain the class ZstdOutputStreamNoFinalizer，and paimon require zstd-jni-1.5.5-11.jar I try to shade zstd-jni https://github.com/apache/paimon/issues/4531,but failed,I think it is a bug

FrommyMind commented 32 minutes ago

Update:

I finally discovered that MR utilizes /user/yarn/mapreduce/mr-framework/3.0.0-cdh6.3.2-mr-framework.tar.gz on HDFS as the location for all dependency jars, so I updated this file to include zstd-jni-1.5.5-1.jar. After making this change, the insert statement executed successfully! Additionally, I ran the original MR job with zstd compression, and it worked perfectly as well.

Perhaps we should consider adding more test cases to ensure that there are no issues with the original MR job. However, for now, everything seems fine to me.