[FEA] Support time zones that are not UTC

revans2 commented 2 years ago

Is your feature request related to a problem? Please describe. This is related to #6603 but a super set of it. I will list out the issues here to complete both sets of work, and then reference some of them in #6603 too.

The following are the tasks to do:

Phase I (P0) - Release 23.12

[x] https://github.com/NVIDIA/spark-rapids/issues/6831
[x] https://github.com/NVIDIA/spark-rapids/issues/9737
[x] https://github.com/NVIDIA/spark-rapids/issues/6832
[x] https://github.com/NVIDIA/spark-rapids/issues/9802
[x] https://github.com/NVIDIA/spark-rapids/issues/9805
[X] https://github.com/NVIDIA/spark-rapids/issues/9804
[x] https://github.com/NVIDIA/spark-rapids/issues/9829 (might move to Phase II)
[x] https://github.com/NVIDIA/spark-rapids/issues/9915 (might move to Phase II)

Phase II (P0) - Release 24.02

[x] https://github.com/NVIDIA/spark-rapids/issues/9606
[x] https://github.com/NVIDIA/spark-rapids/issues/9815
[x] https://github.com/NVIDIA/spark-rapids/issues/9605
[x] https://github.com/NVIDIA/spark-rapids/issues/6836
[x] https://github.com/NVIDIA/spark-rapids/issues/9927
[ ] https://github.com/NVIDIA/spark-rapids/issues/10079
[x] https://github.com/NVIDIA/spark-rapids/issues/9801
[x] https://github.com/NVIDIA/spark-rapids/issues/6834
[x] https://github.com/NVIDIA/spark-rapids/issues/9747
[x] https://github.com/NVIDIA/spark-rapids/issues/9881
[x] https://github.com/NVIDIA/spark-rapids/issues/9855
[ ] https://github.com/NVIDIA/spark-rapids/issues/9633
[ ] https://github.com/NVIDIA/spark-rapids/issues/6840
[ ] https://github.com/NVIDIA/spark-rapids/issues/9997 2 kernel: Parse string to local time; convert local time to UTC time.
[ ] https://github.com/NVIDIA/spark-rapids/issues/10034
[ ] https://github.com/NVIDIA/spark-rapids/issues/10035
[ ] https://github.com/NVIDIA/spark-rapids/issues/10083

Phase III/Uncategorized (P1)

Uncategorized

[ ] https://github.com/NVIDIA/spark-rapids/issues/9653
[ ] https://github.com/NVIDIA/spark-rapids/issues/9912
[ ] https://github.com/NVIDIA/spark-rapids/issues/9913
[x] https://github.com/NVIDIA/spark-rapids/issues/9914 (might move to Phase II)

Test improvements

[ ] #10039
[ ] #10040
[x] #10041
[ ] #141
[ ] https://github.com/NVIDIA/spark-rapids/issues/10874

Benchmarking

[ ] https://github.com/NVIDIA/spark-rapids/issues/10142

In general we are going to try to have timestamps follow what Spark does and store them in UTC. Then convert the inputs to the desired timezone whenever we need to operate on it in a different time zone.

We should also go through other operators and see if there are things that we can/should support once we can support timezones.

nvliyuan commented 1 year ago

Not sure if FromUTCTimestamp support PST should be a sub task. CC @viadea

res-life commented 11 months ago

Follow on issue: https://github.com/NVIDIA/spark-rapids/issues/9570

revans2 commented 11 months ago

Adding in some issues for testing too

https://github.com/NVIDIA/spark-rapids/issues/9627

https://github.com/NVIDIA/spark-rapids/issues/9633

res-life commented 10 months ago

Test cases statistics:	code snapshot	TZ	passed	skipped	xfailed	xpassed
Before xfail all the impacted cases #9773	UTC	19361	2210	373	283	22227
After xfail all the impacted cases #9773	non-UTC	7589	2214	3719	8613	22135
Introduced fine grained checker from #9719	non-UTC	20173	1411	358	382	22324

res-life commented 7 months ago

Not planing work on this for release 24.04 @sameerz can we move it to next release?

NVIDIA / spark-rapids