Timeseries API call sometimes returns all-zero data when given an area parameter

[corviday]

This bug is intermittent and hard to reproduce, but sometimes passing an area polygon as a parameter to the API results in all-zeroes data. It affects multiple data files and data sets.

Here's a link to an example API call demonstrating the error. )

The results of that call are:

{
"id": "tasmin_mClim_BCCAQv2_bcc-csm1-1-m_historical-rcp45_r1i1p1_19610101-19901231_Canada", 
"data": {
  "1977-01-15T00:00:00Z": 0.0, 
  "1977-02-15T00:00:00Z": 0.0, 
  "1977-03-15T00:00:00Z": 0.0, 
  "1977-04-15T00:00:00Z": 0.0,  
  "1977-05-15T00:00:00Z": 0.0, 
  "1977-06-15T00:00:00Z": 0.0, 
  "1977-07-15T00:00:00Z": 0.0, 
  "1977-08-15T00:00:00Z": 0.0, 
  "1977-09-15T00:00:00Z": 0.0, 
  "1977-10-15T00:00:00Z": 0.0, 
  "1977-11-15T00:00:00Z": 0.0, 
  "1977-12-15T00:00:00Z": 0.0
  }, 
"units": "degC"
}

The same area parameter passed to a call to a different id - tasmax instead of tasmin) also returns all zeroes:

{
"id": "tasmax_mClim_BCCAQv2_bcc-csm1-1-m_historical-rcp45_r1i1p1_19610101-19901231_Canada", 
"data": {
  "1977-01-15T00:00:00Z": 0.0, 
  "1977-02-15T00:00:00Z": 0.0, 
  "1977-03-15T00:00:00Z": 0.0, 
  "1977-04-15T00:00:00Z": 0.0, 
  "1977-05-15T00:00:00Z": 0.0, 
  "1977-06-15T00:00:00Z": 0.0, 
  "1977-07-15T00:00:00Z": 0.0, 
  "1977-08-15T00:00:00Z": 0.0,
  "1977-09-15T00:00:00Z": 0.0, 
  "1977-10-15T00:00:00Z": 0.0, 
  "1977-11-15T00:00:00Z": 0.0, 
  "1977-12-15T00:00:00Z": 0.0
  },
 "units": "degC"
}

Giving this area as a parameter to the third and final data file in the test collection has the same result).

{"id": "pr_mClim_BCCAQv2_bcc-csm1-1-m_historical-rcp45_r1i1p1_19610101-19901231_Canada", "data": {
  "1977-01-15T00:00:00Z": 0.0, 
  "1977-02-15T00:00:00Z": 0.0, 
  "1977-03-15T00:00:00Z": 0.0, 
  "1977-04-15T00:00:00Z": 0.0, 
  "1977-05-15T00:00:00Z": 0.0, 
  "1977-06-15T00:00:00Z": 0.0, 
  "1977-07-15T00:00:00Z": 0.0, 
  "1977-08-15T00:00:00Z": 0.0, 
  "1977-09-15T00:00:00Z": 0.0, 
  "1977-10-15T00:00:00Z": 0.0, 
  "1977-11-15T00:00:00Z": 0.0, 
  "1977-12-15T00:00:00Z": 0.0
  }, 
"units": "kg m-2 d-1"
}

No error message is generated by these API calls.

There's reasonably good evidence that the values are not zero in the files themselves, in that the ncWMS maps created from the same file show colour variation over the affected polygon.

Passing the same area parameter to an identical copy of the climate explorer backend running against a different metadata database and different data files does not reproduce the error):

{
"units": "K", 
"id": "tasmax_mClim_CanESM2_rcp45_r1i1p1_20100101-20391231", 
"data": {
  "2025-01-15T00:00:00Z": 279.19142659505206, 
  "2025-02-15T00:00:00Z": 280.26951090494794, 
  "2025-03-15T00:00:00Z": 280.49456787109375, 
  "2025-04-15T00:00:00Z": 279.8186442057292, 
  "2025-05-15T00:00:00Z": 278.9789225260417, 
  "2025-06-15T00:00:00Z": 278.13330078125, 
  "2025-07-15T00:00:00Z": 277.2965494791667, 
  "2025-08-15T00:00:00Z": 276.41054280598956, 
  "2025-09-15T00:00:00Z": 276.2767740885417, 
  "2025-10-15T00:00:00Z": 276.3365071614583, 
  "2025-11-15T00:00:00Z": 276.72873942057294, 
  "2025-12-15T00:00:00Z": 277.8912353515625
  }
}

[jameshiebert]

It looks possible that the low-level data access code is mixing up the latitude axis. Here are the respective masks for this operation.

Investigating...

[jameshiebert]

Fix going in this branch.

[jameshiebert]

TravisCI runs seem happy, but I have a weird issue where the test suite segfaults on my workstation.

(env36) hiebert@pcic-2004:~/code/climate-explorer-backend$ py.test -v --tb=short -k get_array -k prism_pr_small -k metro_van ce/tests
================================================================= test session starts ==================================================================
platform linux -- Python 3.6.1+, pytest-3.1.3, py-1.4.34, pluggy-0.4.0 -- /home/hiebert/code/climate-explorer-backend/env36/bin/python3.6
cachedir: .cache
rootdir: /home/hiebert/code/climate-explorer-backend, inifile: pytest.ini
collected 88 items 

ce/tests/test_api.py::test_stats[metro_van_10] PASSED
ce/tests/test_api.py::test_timeseries[metro_van_10-file0-tasmax] PASSED
ce/tests/test_api.py::test_timeseries_speed[metro_van_10-file0-tasmax] PASSED
ce/tests/test_api.py::test_timeseries_speed[metro_van_10-file4-tasmin] PASSED
ce/tests/test_geo.py::test_clip_speed[metro_van_10-/home/hiebert/code/climate-explorer-backend/ce/tests/data/cgcm.nc] PASSED
ce/tests/test_geo.py::test_clip_speed[metro_van_10-/home/hiebert/code/climate-explorer-backend/ce/tests/data/anuspline_na.nc] PASSED
ce/tests/test_util.py::test_get_array[metro_van_10-cgcm3] PASSED
ce/tests/test_util.py::test_get_array[metro_van_10-cgcm-tmin] PASSED
ce/tests/test_util.py::test_get_array[metro_van_10-prism_pr_small] Segmentation fault (core dumped)

May be related to this rasterio issue?