search

transportenergy / database

Tools for accessing and maintaining the iTEM model & historical databases
https://transportenergy.rtfd.io
GNU General Public License v3.0
24 stars 8 forks source link

Allow multi-step processing #50

Open khaeru opened 3 years ago

khaeru commented 3 years ago

The initial processing scripts for input data set have a “1-step” design pattern: a single function process() does all the work.

This must be extended to allow multiple layers of processing steps, forming a flowchart, graph, or network of processing steps that can be arbitrarily long. This is necessary because, for instance, the step of harmonizing or joining 2 or more data sets must be kept distinct from the step of cleaning/adapting the input data for each one.

This graph pattern is already implemented by other software, e.g. ixmp.reporting—code should be reused instead of reimplemented.

khaeru commented 3 years ago

This graph pattern is already implemented by other software, e.g. ixmp.reporting

This is now genno.