Closed willu47 closed 5 years ago
To recreate, place the following in tests/data_layer/test_data_store.py:TestDataArray
def test_read_write_data_array_all(self, handler, scenario):
data = np.array([0, 1], dtype=float)
spec = Spec.from_dict(scenario['provides'][0])
da = DataArray(spec, data)
handler.write_scenario_variant_data('mortality.csv', da, 2010)
handler.write_scenario_variant_data('mortality.csv', da, 2015)
actual = handler.read_scenario_variant_data('mortality.csv', spec, timestep=None)
expected = np.array([[0, 1], [0, 1]])
np.testing.assert_array_equal(actual.as_ndarray(), expected)
This could be worked around on the usage side - add timestep
with the expected timesteps as a dimension to the spec-to-read.
A slightly-magic-but-nice option could be for read_scenario_variant_data
with timestep=None
to add timestep
as a dimension to the DataArray that's returned, with dimension coordinates as present in the data (e.g. sorted(df.timestep.unique())
).
Does the data store need to support writing partial (i.e. single-timestep) data to a scenario variant? This is implied by the two calls to write_scenario_variant_data
in the snippet above. Current implementations assume that calls to write_scenario_variant_data
are independent and can safely override anything that was previously present in the store.
def test_read_write_data_array_all(self, handler, scenario):
spec = Spec.from_dict(deepcopy(scenario['provides'][0]))
spec_with_t = scenario['provides'][0]
spec_with_t['dims'].insert(0, 'timestep')
spec_with_t['coords']['timestep'] = [2010, 2015]
spec_with_t = Spec.from_dict(spec_with_t)
da = DataArray(spec_with_t, np.array([[0, 1], [2, 3]], dtype='float'))
handler.write_scenario_variant_data('mortality.csv', da)
actual = handler.read_scenario_variant_data('mortality.csv', spec_with_t)
expected = np.array([[0, 1], [2, 3]], dtype='float')
np.testing.assert_array_equal(actual.as_ndarray(), expected)
da_2010 = handler.read_scenario_variant_data('mortality.csv', spec, 2010)
expected = np.array([0, 1], dtype='float')
np.testing.assert_array_equal(da_2010.as_ndarray(), expected)
da_2015 = handler.read_scenario_variant_data('mortality.csv', spec, 2015)
expected = np.array([2, 3], dtype='float')
np.testing.assert_array_equal(da_2015.as_ndarray(), expected)
Output as dataframe:
mortality
timestep lad
2010 a 0.0
b 1.0
2015 a 2.0
b 3.0
2010:
mortality
lad
a 0.0
b 1.0
2015:
mortality
lad
a 2.0
b 3.0
This has been addressed in #373 (and associated PR #375) but the creation of a new method on the Store
called read_scenario_variant_data_multiple_timesteps()
and corresponding method on Results
called read_scenario_data()
.
Note that there is no way of providing timesteps=None
get get all available data, because the spec for the resulting DataFrame
needs to be known, and timesteps are a property only of the model run.
I have now updated the original method read_scenario_variant_data()
to take timestep
as a non-optional argument.
This can now be closed as the original call (with timestep=None
) is no longer permitted.
Passing
timestep=None
to to Store.read_scenario_variant raises a SmifMismatchError, but it should return a DataArray with timestep as a dimension of the Spec.