Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
If you have triaged this as a bug, then file an issue directly.
Describe the problem you faced
Hi Everyone, I have a couple of questions using hudi table with different versions of EMR. I have existing table hudi table with EMR version: 6.4 with table version as 1 and when i used EMR version 6.12, the hudi table got upgraded to table version as 5. When I tried to write with the same partition field. It
get this below error when i try to run hudi upsert emr step using 6.4
py4j.protocol.Py4JJavaError: An error occurred while calling o399.save.
: org.apache.hudi.exception.HoodieException: Unknown versionCode:5
Tried the same with EMR 6.12 it provided below error where in my case partition_column has not changed.
org.apache.hudi.exception.HoodieException: Config conflict(key current value existing value):
PartitionPath: col_1:SIMPLE col_1
at org.apache.hudi.HoodieWriterUtils$.validateTableConfig(HoodieWriterUtils.scala:211)
at org.apache.hudi.HoodieSparkSqlWriter$.writeInternal(HoodieSparkSqlWriter.scala:177)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:132)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)
I could not able to downgrade the hudi version so that it goes to table_version 1. I want even though i haven’t initiated any upgrade of hudi version how table version updated with new version of EMR 6.12 but while writing it got stuck.
Do i need to make any changes so that it uses latest hudi_table_version:5 ? Any help is appreciated.
A clear and concise description of the problem.
To Reproduce
Steps to reproduce the behavior:
1.
2.
3.
4.
Expected behavior
A clear and concise description of what you expected to happen.
Tips before filing an issue
Have you gone through our FAQs?
Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
If you have triaged this as a bug, then file an issue directly.
Describe the problem you faced Hi Everyone, I have a couple of questions using hudi table with different versions of EMR. I have existing table hudi table with EMR version: 6.4 with table version as 1 and when i used EMR version 6.12, the hudi table got upgraded to table version as 5. When I tried to write with the same partition field. It get this below error when i try to run hudi upsert emr step using 6.4 py4j.protocol.Py4JJavaError: An error occurred while calling o399.save. : org.apache.hudi.exception.HoodieException: Unknown versionCode:5 Tried the same with EMR 6.12 it provided below error where in my case partition_column has not changed. org.apache.hudi.exception.HoodieException: Config conflict(key current value existing value): PartitionPath: col_1:SIMPLE col_1 at org.apache.hudi.HoodieWriterUtils$.validateTableConfig(HoodieWriterUtils.scala:211) at org.apache.hudi.HoodieSparkSqlWriter$.writeInternal(HoodieSparkSqlWriter.scala:177) at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:132) at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150) at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90) I could not able to downgrade the hudi version so that it goes to table_version 1. I want even though i haven’t initiated any upgrade of hudi version how table version updated with new version of EMR 6.12 but while writing it got stuck. Do i need to make any changes so that it uses latest hudi_table_version:5 ? Any help is appreciated. A clear and concise description of the problem.
To Reproduce
Steps to reproduce the behavior:
1. 2. 3. 4.
Expected behavior
A clear and concise description of what you expected to happen.
Environment Description
Hudi version :
Spark version :
Hive version :
Hadoop version :
Storage (HDFS/S3/GCS..) :
Running on Docker? (yes/no) :
Additional context
Add any other context about the problem here.
Stacktrace
Add the stacktrace of the error.