search

apache / iceberg

Apache Iceberg
https://iceberg.apache.org/
Apache License 2.0
6.24k stars 2.17k forks source link

Cannot find constructor for interface org.apache.parquet.column.page.PageWriteStore? #9802

Open williamliu111 opened 7 months ago

williamliu111 commented 7 months ago

Query engine

No response

Question

use iceberg java api to write data, The error message is as follows：

Exception in thread "main" java.lang.ExceptionInInitializerError at org.apache.iceberg.parquet.Parquet$WriteBuilder.build(Parquet.java:268) at Iceberg.writeFile(Iceberg.java:113) at Iceberg.main(Iceberg.java:200) Caused by: java.lang.RuntimeException: Cannot find constructor for interface org.apache.parquet.column.page.PageWriteStore Missing org.apache.parquet.hadoop.ColumnChunkPageWriteStore(org.apache.parquet.hadoop.CodecFactory$BytesCompressor,org.apache.parquet.schema.MessageType,org.apache.parquet.bytes.ByteBufferAllocator,int) [java.lang.NoSuchMethodException: org.apache.parquet.hadoop.ColumnChunkPageWriteStore.(org.apache.parquet.hadoop.CodecFactory$BytesCompressor, org.apache.parquet.schema.MessageType, org.apache.parquet.bytes.ByteBufferAllocator, int)] at org.apache.iceberg.common.DynConstructors$Builder.build(DynConstructors.java:236) at org.apache.iceberg.parquet.ParquetWriter.(ParquetWriter.java:58) ... 3 more

my part pom.xml:

org.apache.parquet parquet-column 1.12.0 
<dependency>
    <groupId>org.apache.parquet</groupId>
    <artifactId>parquet-jackson</artifactId>
    <version>1.12.0</version>
</dependency>
<dependency>
    <groupId>org.apache.parquet</groupId>
    <artifactId>parquet-hadoop</artifactId>
    <version>1.12.0</version>
</dependency>
<dependency>
    <groupId>org.apache.parquet</groupId>
    <artifactId>parquet-avro</artifactId>
    <version>1.11.1</version>
</dependency>
<!-- parquet-common增加bzip2、deflate压缩格式 -->
<dependency>
    <groupId>org.apache.parquet</groupId>
    <artifactId>parquet-common</artifactId>
    <version>1.12.0</version>
</dependency>
<dependency>
    <groupId>org.apache.parquet</groupId>
    <artifactId>parquet-format-structures</artifactId>
    <version>1.12.0</version>
</dependency>
<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-storage-api</artifactId>
    <version>2.7.3</version>
</dependency>
<dependency>
    <groupId>org.apache.avro</groupId>
    <artifactId>avro</artifactId>
    <version>1.9.0</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.12</artifactId>
    <version>3.3.0</version>
</dependency>
manuzhang commented 7 months ago

It looks you have two versions of parquet in your dependencies, 1.11.1 and 1.12.0