nismod / smif

Simulation Modelling Integration Framework
http://www.itrc.org.uk
MIT License
22 stars 6 forks source link

results - Results.read_scenario_data reads in incorrect data #378

Closed willu47 closed 5 years ago

willu47 commented 5 years ago

results.read_scenario_data reads in the wrong data, and the data changes as the list of timesteps passed changes. Somewhere, there is a misalignment between the Spec coordinates and the data.

For example, actual data:

timestep lad_uk_2016 residential_or_non floor_area
2030 E06000056 residential 10841750.0
2030 E06000056 non_residential 6525.1228990099

Using the api

>>> df = results.read_scenario_data(
        'socio-economic', 'arc_baseline', 'floor_area', [2030])
>>> df[df['lad_uk_2016'] == 'E06000056']

returns the incorrected data:

timestep lad_uk_2016 residential_or_non floor_area
2030 E06000056 residential 3424711.285
2030 E06000056 non_residential 4647666.550

Direct read from the store gives expected data:

>>> store = results._store
>>> df = store.read_scenario_variant_data('socio-economic', 'arc_baseline', 'floor_area', 2030).as_df().reset_index()
>>> df[df['lad_uk_2016'] == 'E06000056']
lad_uk_2016 residential_or_non floor_area
E06000056 residential 1.084175e+07
E06000056 non_residential 6.525123e+03
fcooper8472 commented 5 years ago

Associated changes also needed in the results API, which is now more thoroughly tested.