search

Netflix / iceberg

Iceberg is a table format for large, slow-moving tabular data
Apache License 2.0
476 stars 59 forks source link

java.lang.AbstractMethodError: Method org/apache/iceberg/spark/source/SparkTable.newWriteBuilder #117

Closed Arnaud-Nauwynck closed 3 years ago

Arnaud-Nauwynck commented 3 years ago

Hello,

I encountered the following error while testing iceberg locally

I use spark-3.0.0-preview-bin-hadoop3.2 + hadoop-3.2.1

bin\spark-shell --packages org.apache.iceberg:iceberg-spark3-runtime:0.9.1 \ --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \ --conf spark.sql.catalog.spark_catalog.type=hive \ --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \ --conf spark.sql.catalog.local.type=hadoop \ --conf spark.sql.catalog.local.warehouse=$PWD/warehouse

scala> spark.sql("CREATE TABLE db1.iceberg_table2 (id bigint, data string) USING iceberg LOCATION 'file:/D:/arn/hadoop/rootfs/db1/iceberg_table2'").show()

.. ok, then insert error:

scala> spark.sql("INSERT INTO db1.iceberg_table2 VALUES (1, 'a'), (2, 'b'), (3, 'c')").show() 2020-10-12 20:53:11,992 INFO metastore.HiveMetaStore: 0: get_database: global_temp 2020-10-12 20:53:11,992 INFO HiveMetaStore.audit: ugi=arnaud ip=unknown-ip-addr cmd=get_database: global_temp 2020-10-12 20:53:11,995 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException 2020-10-12 20:53:12,005 INFO metastore.HiveMetaStore: 0: get_table : db=db1 tbl=iceberg_table2 2020-10-12 20:53:12,005 INFO HiveMetaStore.audit: ugi=arnaud ip=unknown-ip-addr cmd=get_table : db=db1 tbl=iceberg_table2 2020-10-12 20:53:12,058 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: file:/D:/arn/hadoop/rootfs/db1/iceberg_table2/metadata/00000-0398ba31-39af-4ec6-a0ee-81a79c1bdabb.metadata.json java.lang.AbstractMethodError: Method org/apache/iceberg/spark/source/SparkTable.newWriteBuilder(Lorg/apache/spark/sql/util/CaseInsensitiveStringMap;)Lorg/apache/spark/sql/connector/write/WriteBuilder; is abstract at org.apache.iceberg.spark.source.SparkTable.newWriteBuilder(SparkTable.java) at org.apache.spark.sql.execution.datasources.v2.BatchWriteHelper.newWriteBuilder(WriteToDataSourceV2Exec.scala:338) at org.apache.spark.sql.execution.datasources.v2.BatchWriteHelper.newWriteBuilder$(WriteToDataSourceV2Exec.scala:337) at org.apache.spark.sql.execution.datasources.v2.AppendDataExec.newWriteBuilder(WriteToDataSourceV2Exec.scala:249) at org.apache.spark.sql.execution.datasources.v2.AppendDataExec.doExecute(WriteToDataSourceV2Exec.scala:255) at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:189) at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:227) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:224) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:185) at org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:329) at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:378) at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:226) at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3407) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$4(SQLExecution.scala:100) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:87) at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3403) at org.apache.spark.sql.Dataset.(Dataset.scala:226) at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:607) ... 47 elided

Arnaud-Nauwynck commented 3 years ago

I have tested with spark-3.0.1-bin-hadoop3.2 + hadoop-3.2.1 It works .. closing this issue