search

calliope-project / calliope

A multi-scale energy systems modelling framework
https://www.callio.pe
Apache License 2.0
299 stars 93 forks source link

Masking timeseries causes clustering and NetCDF save errors #168

Closed brynpickering closed 3 years ago

brynpickering commented 5 years ago

Problem description

  1. If masking a timeseries followed by clustering, the timeseries given to the clustering function may be an incorrect length, particularly if the mask includes the last timestep in the timeseries

  2. For each mask, the information about the chosen timesteps in the mask is saved to the model xarray Dataset attributes. As it is saved as a pandas DataFrame, it cannot be serialised for saving to NetCDF. Either the information has to be removed, or has to be simplified to a string.

  3. Since the introduction of datesteps (for inter-cluster storage), masking needs to be updated to include its own datesteps in the final datestep list.

This all needs updating, plus tests to cover these combinations added.

Steps to reproduce the problem

Applying timeseries clustering e.g.:

    model.time:
        masks:
            - {function: extreme_diff, options: {tech0: pv, tech1: demand_electricity, how: max}}
            - {function: extreme, options: {tech: demand_electricity, how: max}}
        function: apply_clustering
        function_options: {clustering_func: kmeans, how: mean, k: 2}

Calliope version

0.6.3

brynpickering commented 3 years ago

Closed as it may(?) not be an issue any more and we are anyway moving timeseries aggregation out of Calliope's core functionality (#356)