NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
797 stars 232 forks source link

[FEA] [Follow on] Follow on issues for the checking non UTC test cases. #9627

Open res-life opened 11 months ago

res-life commented 11 months ago

Describe the bug

After https://github.com/NVIDIA/spark-rapids/pull/9482 is merged, should further finish the following items:

import os
def get_time_zone_for_testing():
  tz = os.getenv('TEST_SPARK_SESSION_TIMEZONE')
  if tz is None:
    tz = "UTC"
  else:
    tz = tz.strip()
  return tz

# spark_session.py
tz = get_time_zone_for_testing()
_spark.conf.set("spark.sql.session.timeZone", tz)

# conftest.py
    if (item.get_closest_marker('disable_timezone_test') and 
        get_time_zone_for_testing() != "UTC"):
        // for cases disable non-utc test and current timezone for testing is non-utc, skip them.
        pytest.skip('Skip because this case is not ready for non UTC time zone')

Additional context Finally we should remove the fallback cases that testing non UTC.

res-life commented 7 months ago

Not planing work on this for release 24.04, unassign it from me.