quintel / etlocal

Regional data analysis tool for the Energy Transition Model
https://data.energytransitionmodel.com/
MIT License
2 stars 2 forks source link

Incorporate Ruby dataset amalgamator #481

Closed thomas-qah closed 11 months ago

thomas-qah commented 11 months ago

Why?

Up until now the dataset amalgamator (or 'dataset-combiner') was a collection of Python scripts living in a separate branch of the etlocal repository. This posed several challenges in the form of maintainability and interoperability with the rest of the application. A rewrite of these scripts into Ruby addresses these challenges.

What?

This PR rewrites the dataset-combiner from Python to Ruby and thereby integrates it into the existing etlocal Ruby/Rails codebase. Furthermore:

How?

This PR introduces 3 new Ruby classes:

Usage

The script can by used through the command-line. Here's an example:

rails dataset:combine source_dataset_geo_ids=GM306,GM307,GM308 source_data_year=2019
                      target_dataset_geo_id=PV20 target_area_name=Groningen
                      migration_slug=update_2019

To see the full list of arguments (required and optional) for this command, please run:

rails dataset:combine --help

Note

This PR includes the updated interface element yaml files from the dataset-amalgamator branch. It does not include the data migrations from that branch.

Closes #464 Closes #477