Closed dht7 closed 12 months ago
@dht7 Able to reproduce this issue.
Root cause was the source table had column nullable false but the hudi table had it true.
Source table
Target Table
Spark SQL Code -
create database test_database1;
create table test_database1.source_table
as select 1 as id, array(1,2,3) as array_column1;
create table test_database1.test_table using hudi options (type="cow", primaryKey='id', hoodie.table.name="test_table")
select id, array_column1 from test_database1.source_table;
insert overwrite table test_database1.test_table select id, array_column1 FROM test_database1.source_table;
JIRA created to fix this behaviour - https://issues.apache.org/jira/browse/HUDI-6438
Thank you @ad1happy2go for testing the issue and creating the JIRA ticket.
Just wanted to add some additional details:
The source table in my case has the columns as nullable (on printSchema for table df in spark):
But, the avro schema does have them as non-nullable, i.e.
{
"type" : "record",
"name" : "test_table_record",
"namespace" : "hoodie.test_table",
"fields" : [ {
"name" : "id",
"type" : "int"
}, {
"name" : "array_column1",
"type" : [ "null", {
"type" : "array",
"items" : "int"
} ]
} ]
}
@ad1happy2go I am not able to reproduce the issue against the latest master version of hudi using either spark-3.1 and spark-3.2 using the steps you outlined. Do we know if this issue is limited only to older version of Hudi (version : 0.12.2 as reported in the description)?
yes , I also confirmed with master I am not seeing this issue.
@dht7 Can you check with master code if possible if you are still facing this issue?
The issue was fixed through PR: #8725. Error message shows up before this PR.
Thank you @ad1happy2go I was able to test this and can confirm that we are not facing the issue when using the latest master code.
Thank you @amrishlal for pointing this out and sharing the PR details.
cc: @codope
Hi @ad1happy2go ,
Using Hudi : 0.14.1
With this change - is it fair to say that we can't mitigate the error Incoming batch schema is not compatible with the table's one
?
How to solve for this scenario of schema mismatch since table schema has non-nullable and new data has null values.
Background:
I got the original error of non-nullable column can't accept null values then I removed using MAKE_NEW_COLUMNS_NULLABLE as parameter which was originally configured to support schema evolution, but now I get Incoming batch schema is not compatible with the table's one
since the Hudi table has null for a column.
Describe the problem you faced
Apache Hudi tables created using CTAS command in Spark-SQL with array column types fail to insert overwrite data.
To Reproduce
Steps to reproduce the behavior:
Run the following create statement via Spark-SQL (or Spark-Thrift server):
Here, source_table is also a Hudi table.
Once the table is created, try to update the data by running the following query:
Running the above query results into org.apache.spark.sql.AnalysisException (complete stack-trace attached below).
Expected behavior
Insert overwrite query should get executed on the target table without any issues/exceptions.
Environment Description
Hudi version : 0.12.2
Spark version : 3.1.2
Storage (HDFS/S3/GCS..) : GCS
Running on Docker? (yes/no) : no
Additional context
Stacktrace