openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference
MIT License
13 stars 11 forks source link

Test fixes - timezones #193

Closed dfulu closed 1 year ago

dfulu commented 1 year ago

Pull Request

Overview

  1. Some of the tests were failing due to timezone incompatibilities. Removing the explicit timezones will fix them.
  2. Update SelectLiveT0TimeIterDataPipe due to deprecation
  3. Update test broken by GSP boundary area fix.
  4. Missing import.

Detail

  1. Most, if not all of the datasets we use store the datetime as np.datetime64 which, as far as I can gather, is unaware of timezones and instead is measured a seconds since the UNIX epoch.

The tests where we slice xarray datasets using datetime.datetime(..., tzinfo=datetime.timezone.utc) where we explicitly state the timezone were causing an error such as :

TypeError: Cannot compare tz-naive and tz-aware datetime-like objects.

The fix is to remove the explicit timezone info for the slicing. As far as I am aware, all of our datasets are expressed in UTC, so this should not be a problem.

codecov[bot] commented 1 year ago

Codecov Report

Merging #193 (c82257f) into main (2263cd7) will increase coverage by 0.34%. The diff coverage is 93.33%.

@@            Coverage Diff             @@
##             main     #193      +/-   ##
==========================================
+ Coverage   81.92%   82.27%   +0.34%     
==========================================
  Files         124      127       +3     
  Lines        5102     5268     +166     
==========================================
+ Hits         4180     4334     +154     
- Misses        922      934      +12     
Impacted Files Coverage Δ
ocf_datapipes/training/common.py 95.54% <86.79%> (-4.46%) :arrow_down:
ocf_datapipes/transform/xarray/standard_dropout.py 94.11% <94.11%> (ø)
ocf_datapipes/select/select_loc_and_t0.py 95.65% <95.65%> (ø)
ocf_datapipes/transform/xarray/nwp_dropout.py 97.56% <97.56%> (ø)
ocf_datapipes/load/gsp/utils.py 100.00% <100.00%> (ø)
ocf_datapipes/select/__init__.py 100.00% <100.00%> (ø)
ocf_datapipes/select/select_live_t0_time.py 100.00% <100.00%> (ø)
ocf_datapipes/transform/xarray/__init__.py 100.00% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more