search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.43k stars 955 forks source link

[core] Fix merge schemas equal method issue #4482

Closed harveyyue closed 1 week ago

harveyyue commented 2 weeks ago

Purpose

Linked issue: close #xxx No

Tests

API and Format

Documentation

wwj6591812 commented 1 week ago

+1

JingsongLi commented 1 week ago

I remember this is not an issue. @harveyyue Can you find the exact case?

harveyyue commented 1 week ago

I remember this is not an issue. @harveyyue Can you find the exact case?

Using spark format paimon to write upstream data will generate a new schema-xxx since have no schema changes dataset.format("paimon").options(options);

JingsongLi commented 1 week ago

I remember this is not an issue. @harveyyue Can you find the exact case?

Using spark format paimon to write upstream data will generate a new schema-xxx since have no schema changes dataset.format("paimon").options(options);

@harveyyue Can you add a test case?

harveyyue commented 1 week ago

I remember this is not an issue. @harveyyue Can you find the exact case?

Using spark format paimon to write upstream data will generate a new schema-xxx since have no schema changes dataset.format("paimon").options(options);

@harveyyue Can you add a test case?

Done We did some code changes to reselect column according table schema, the unexecpted field orders will trigger the == compare method from row type.