csiro-hydroinformatics / pybomwater

Tool for requesting data from BoM Water Data Service
Other
21 stars 2 forks source link

IndexError: list index out of range on a query for water temperature that appears legit #10

Open jmp75 opened 2 months ago

jmp75 commented 2 months ago

Noticed using the version on the main branch as I write, i.e this commit

Observations

The method values_count appears to expect at least one observation to be available querying the response; we hit a case where there is none.

Reproduction

import pybomwater.bom_water as bm
import os

bm = bm.BomWater()
procedure = bm.procedures.Pat4_C_B_1_DailyMean
prop = bm.properties.Water_Temperature

# Setup bounding box for BoM api spatial filter query
low_left_lat = -37.505032
low_left_long = 138.00
upper_right_lat = -24.00
upper_right_long = 154.00

lower_left_coords = f'{low_left_lat} {low_left_long}'
upper_right_coords = f'{upper_right_lat} {upper_right_long}'
coords = tuple((lower_left_coords, upper_right_coords))

t_begin = "1800-01-01T00:00:00+10"
t_end = "2030-12-31T00:00:00+10"

from pathlib import Path
spat_dir = Path(os.path.expanduser('~')) / "src" / "bomwater-notebook" / "bomwater_data" / "Spatial"
spatial_path = spat_dir / 'mdb_buffer_1km.shp'
assert spatial_path.exists()

results = bm.get_spatially_filtered_observations( None, str(spatial_path), coords, prop, procedure, t_begin, t_end)

There may be a random aspect in values_count, but this stilll seems to lead consistently to:

Exception has occurred: IndexError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
list index out of range
  File "/home/xyzxyz/src/pybomwater/pybomwater/bom_water.py", line 418, in values_count
    value_nodes = observations[0].findall(query_measurement)
                  ~~~~~~~~~~~~^^^
  File "/home/xyzxyz/src/pybomwater/pybomwater/bom_water.py", line 332, in define_request_chunking_size
    values_count = self.values_count(response)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xyzxyz/src/pybomwater/pybomwater/bom_water.py", line 355, in batch_request_observations
    chunk_size = self.define_request_chunking_size( station_no, property, procedure, t_begin, t_end )
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xyzxyz/src/pybomwater/pybomwater/bom_water.py", line 348, in get_spatially_filtered_observations
    return self.batch_request_observations(property, procedure, t_begin, t_end, filtered_features)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xyzxyz/src/a2s-mdb/notebooks/bom_water_temp_downloader.py", line 69, in <module>
    results = bm.get_spatially_filtered_observations( None, str(spatial_path), coords, prop, procedure, t_begin, t_end)

IndexError: list index out of range

This appears to fail for feature 'http://bom.gov.au/waterdata/services/stations/425012'

fre171csiro commented 2 months ago

I implemented you code as a test and resolved this now. @jmp75 can you please check if this is correct?