Hi everyone, i'm trying to test schema evolution for my cdc pipline (debizium + kafka) with hudi 0.11.0 and spark structured streming , i follow this documentation, https://hudi.apache.org/docs/0.11.0/schema_evolution,
does hudi manage well the schema evolution, it is necessary to restart the job, once it is done all the old values will be null, it takes into account only the values of the last commits, so the data not match my Postgres source ? on my schema registry i can see V1 and V2
any ideas thanks
A clear and concise description of the problem.
To Reproduce
Steps to reproduce the behavior:
Stop hudi streams, and drop hive tables
add some columns ALTER TABLE
ADD COLUMN character varying(50) DEFAULT 'toto' ;
restart hudi spark jobs
select * from hudi _ro / _rt table ( or read parquet hudi format using spark)
Expected behavior
when i select my data it expected to see default value on the added column and not null values
data on postgres source:
cdc_hudi=> select test, test2, test3 from hudipart
;
test | test2 | test3
------+-------+-------
toto | f | Toto
test | t | Toto
test | t | Toto
test | t | Toto
toto | f | Toto
toto | f | Toto
toto | f | Toto
toto | f | Toto
toto | f | Toto
toto | f | Toto
toto | f | Toto
test | t | Toto
test | t | Toto
test | t | test3
test | t | test3
toto | f | Toto
toto | f | Toto
toto | f | Toto
test | t | test3
toto | f | Toto
toto | f | Toto
toto | f | Toto
Hi everyone, i'm trying to test schema evolution for my cdc pipline (debizium + kafka) with hudi 0.11.0 and spark structured streming , i follow this documentation, https://hudi.apache.org/docs/0.11.0/schema_evolution, does hudi manage well the schema evolution, it is necessary to restart the job, once it is done all the old values will be null, it takes into account only the values of the last commits, so the data not match my Postgres source ? on my schema registry i can see V1 and V2
any ideas thanks
A clear and concise description of the problem.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
when i select my data it expected to see default value on the added column and not null values
data on postgres source:
data on hudi parquets / hive tables :
Environment Description
Hudi version : 0.11.0
Spark version :3.1.4
Hive version :1.2.1000
Hadoop version : 2.7.3
Storage (HDFS)
schema registry
Kafka
debezium
@Armelabdelkbir spark now not support default value, maybe https://github.com/apache/spark/pull/36672/files can help you, thanks
@Armelabdelkbir if you has this requirement for spark 3.1x pls raise a pr, and i will fixed it as soon as possible
Closing as the issue has been triaged and we have a patch as suggested in https://github.com/apache/hudi/issues/6496#issuecomment-1228023484