zhouyuan commented 9 months ago

Backend

VL (Velox)

Bug description

There are several data mismatch issues either related with operator or functions. Some of the fixes are landed in Gluten, and some are in Velox repo. We will use this issue to track the status as these are critical for production envs.

[x] complex datatype return wrong value, disabled in Gluten now
[x] parquet scan + filter pushdown wrongly return "", should return null.
[x] distinct hash agg + spill returned duplicated keys.
[x] max_by function return wrong result
[x] sortBeforeRepartition
[ ] cast(sum(decmial(20,4)), float)
[x] get_json_object({"dScore":0.0215434648799772}, "$.dScore") (Fixed in Velox)
[ ] cast(string as bigint)
[x] cast(double as decimal)
[ ] array_size(null) (issue: https://github.com/apache/incubator-gluten/issues/5248)
[x] round(avg(cast(col as double)), 4) #5366
[x] isNull and isNotNull in filter condition #5670
[x] from_unixtime with overflowed (Fixed by https://github.com/facebookincubator/velox/pull/9836)
[ ] date_format (https://github.com/apache/incubator-gluten/issues/5524)
[x] cast integer as binary (https://github.com/apache/incubator-gluten/issues/5073)
[ ] regexp_replace('a{bc', '\{', '\[') (#6224)
[ ] LEGACY timeParserPolicy (#6227)
[ ] FlushableAgg (#6630)
[ ] if expression (#6673)
[ ] weekOfYear (#6784)
[ ] round (#6827)
[ ] cast string as date (#6828)
[ ] nested decimal arithmetic expressions (#7082)
[ ] date format week year (#7069)
[ ] Large timestamp outside of range (#7109)
[ ] Aggregate window gets the wrong result (#7194)
[x] Diff of in_or_and (#7362)
[ ] hash agg output wrong result (#7494)
[ ] cast double as string (#7730)
[ ] Cast string to integral type does not ignore ISO control characters (#7749)
[ ] get_json_object (#7777)

FelixYBW commented 9 months ago

4678 issue in hashagg

FelixYBW commented 9 months ago

https://github.com/oap-project/gluten/issues/4587

Currently we disabled all complex data read

zhouyuan commented 8 months ago

https://github.com/oap-project/gluten/pull/4818

zhouyuan commented 8 months ago

https://github.com/oap-project/gluten/pull/4872

kecookier commented 8 months ago

https://github.com/apache/incubator-gluten/issues/4891

kecookier commented 8 months ago

https://github.com/apache/incubator-gluten/issues/4928

kecookier commented 8 months ago

https://github.com/apache/incubator-gluten/issues/4930

kecookier commented 8 months ago

https://github.com/apache/incubator-gluten/issues/4947

FelixYBW commented 8 months ago

3 issues we met:

parquet scan + filter pushdown wrongly return "", should return null. Fixed by https://github.com/facebookincubator/velox/pull/9129
distinct hash agg + spill returned duplicated keys. https://github.com/facebookincubator/velox/issues/9219
max_by function return wrong result

ulysses-you commented 8 months ago

distinct hash agg + spill returned duplicated keys.

@FelixYBW Has this issue not been fixed by https://github.com/apache/incubator-gluten/pull/4443 ?

FelixYBW commented 8 months ago

@FelixYBW Has this issue not been fixed by #4443 ?

No, it's tested from main branch. A new issue

FelixYBW commented 8 months ago

No, it's tested from main branch. A new issue

https://github.com/facebookincubator/velox/issues/9219

FelixYBW commented 8 months ago

max_by function return wrong result

@yma11 Did you submit a fix to the issue?

yma11 commented 8 months ago

max_by function return wrong result

@yma11 Did you submit a fix to the issue?

Not yet. Only have pushed to golden branch and will submit one in Velox upstream.

NEUpanning commented 7 months ago

5253

FelixYBW commented 7 months ago

5253

Looks the issue of get_json_object. @PHILO-HE maybe we need a fully tests of json functions, like the re2.

PHILO-HE commented 7 months ago

5253

Looks the issue of get_json_object. @PHILO-HE maybe we need a fully tests of json functions, like the re2.

@FelixYBW, I will do that. Thanks!

FelixYBW commented 7 months ago

https://github.com/apache/incubator-gluten/issues/5248

kecookier commented 7 months ago

https://github.com/apache/incubator-gluten/issues/5366

FelixYBW commented 7 months ago

5366

UPdated desc. thank you. do you know which function (cast, avg, round ) caused the issue?

FelixYBW commented 7 months ago

5372

yma11 commented 7 months ago

max_by function return wrong result

@yma11 Did you submit a fix to the issue?

Not yet. Only have pushed to golden branch and will submit one in Velox upstream.

@FelixYBW This fix should be done at cpp side. The formal fix is in PR. Can you help review it?

FelixYBW commented 7 months ago

@FelixYBW This fix should be done at cpp side. The formal fix is in PR. Can you help review it?

Is it a Gluten issue? I'd think veox has some bug here.

yma11 commented 7 months ago

@FelixYBW This fix should be done at cpp side. The formal fix is in PR. Can you help review it?

Is it a Gluten issue? I'd think veox has some bug here.

Yes. I think so. It's caused by the additional projects we added before/after shuffle. The logic of partial/final handle in Velox upstream has no problem. The ideal way is to add struct support for shuffle in Gluten so that we can remove the hack.

FelixYBW commented 6 months ago

@PHILO-HE Any update of the issues here?