[Spark] Support duplicate keys scenario for transform_keys function

facebookincubator / velox

A C++ vectorized database acceleration library aimed to optimizing query engines and data processing systems.

https://velox-lib.io/

Apache License 2.0

3.46k stars 1.13k forks source link

[Spark] Support duplicate keys scenario for transform_keys function #10219

Open gaoyangxiaozhu opened 3 months ago

gaoyangxiaozhu commented 3 months ago

Description

current implement of transform_keys always throw exception if duplicate keys exists after transform, while for spark, if the conf MAP_KEY_DEDUP_POLICY be set to SQLConf.MapKeyDedupPolicy.LAST_WIN, then instead of throw exception, the duplicated map keys will be removed w.r.t. the last wins policy

PHILO-HE commented 3 months ago

@gaoyangxiaozhu, seems a PR is fixing this issue: https://github.com/facebookincubator/velox/pull/9562. Please take a review.

gaoyangxiaozhu commented 3 months ago

@gaoyangxiaozhu, seems a PR is fixing this issue: #9562. Please take a review.

thanks, @PHILO-HE , let's wait the fix ready