search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.13k stars 842 forks source link

[spark] Fix UpdateCommond Compatibility in non-primary key #3561

Closed xuzifu666 closed 2 weeks ago

xuzifu666 commented 2 weeks ago

Can not update data from non-pk table

java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation$.apply$default$4()Lscala/Option; at org.apache.paimon.spark.commands.UpdatePaimonTableCommand.performUpdateForNonPkTable(UpdatePaimonTableCommand.scala:110) at org.apache.paimon.spark.commands.UpdatePaimonTableCommand.run(UpdatePaimonTableCommand.scala:60) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:80) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:78) at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:89) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:110) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:110) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:106) at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)

Anything else?

Linked issue: https://github.com/apache/paimon/issues/3560

Tests

API and Format

Documentation

JingsongLi commented 2 weeks ago

@xuzifu666 can you rebase latest master?

xuzifu666 commented 2 weeks ago

@xuzifu666 can you rebase latest master?

https://github.com/apache/paimon/pull/3566 The pr conflict to this pr which merge early today and content is close to this pr. @JingsongLi

Zouxxyy commented 2 weeks ago

@xuzifu666 Very sorry for that, I didn't see this PR before, thank you for your support of paimon spark, I was so careless, my deepest apologies to you

xuzifu666 commented 2 weeks ago

@xuzifu666 Very sorry for that, I didn't see this PR before, thank you for your support of paimon spark, I was so careless, my deepest apologies to you

Don't care bro，win-win cooperation ~ @Zouxxyy and I close the pr firstly