search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.13k stars 842 forks source link

[Bug] Tag createTime is null #3556

Closed wForget closed 2 weeks ago

wForget commented 2 weeks ago

Search before asking

Paimon version

0.8.0

Compute Engine

spark

Minimal reproduce step

auto create tag using process-time

image

What doesn't meet your expectations?

Tag createTime should always be set.

Anything else?

No response

Are you willing to submit a PR?

JingsongLi commented 2 weeks ago

Hi @wForget , this is because we need to be compatible to old readers. Very regretful, the JsonIgnoreProperties(ignoreUnknown = true) not be set in Snapshot for old versions.

JingsongLi commented 2 weeks ago

Hi @wForget , I closed #3557, we can discuss here.

wForget commented 2 weeks ago

Hi @wForget , this is because we need to be compatible to old readers. Very regretful, the JsonIgnoreProperties(ignoreUnknown = true) not be set in Snapshot for old versions.

Thanks for the explanation, but I still don't understand why this is blocking, it is always possible to write to the new added tagTimeRetained and tagCreateTime fields.

JingsongLi commented 2 weeks ago

@wForget

JingsongLi commented 2 weeks ago

I add this comments:

// When timeRetained is not defined, please do not write the tagCreatorTime field,
// as this will cause older versions (<= 0.7) of readers to be unable to read this
// tag.
// When timeRetained is defined, it is fine, because timeRetained is the new
// feature.
wForget commented 2 weeks ago

@wForget

  • New writer write new tag, there is a tagCreateTime field.
  • Old reader read this tag, failed, because of json unknown field exception.

Got it, thanks.