rl-institut / oemof-B3

An open-source energy system model for Brandenburg/Berlin.
https://oemof-b3.readthedocs.io/
GNU Affero General Public License v3.0
9 stars 5 forks source link

Added script to download resources #288

Closed henhuy closed 1 year ago

henhuy commented 1 year ago

Added script download_resources which downloads resources from OEP and saves them in folder results/_resources. TODOS:

@jnnr could you complete TODOS above, as you might know better all resources and related tables on OEP. and you are more familiar with snakemake...?

henhuy commented 1 year ago

Download of raw_data could be implemented easily by using function download_table_from_OEP from oemof_b3.tools.oep

jnnr commented 1 year ago

Downloading raw data is much easier to implement into the pipeline than resources. Something like:

rule get_raw_data:
    outputs: "raw/..."
    scripts: 
        if config.raw.source == "oep"
            download_from_oep
        elif config.raw.source == "zenodo"
            download_from_zenodo
        elif config.raw.source == "manual"
            logger.warning("Have to provide raw data manually by copying it to raw")

This is because there is no rule for raw data yet, but there are many rules for creating resources.

Idea: Implement a switch in Snakefile that allows to select which resources should be used, either results/_resources or results/_oep_resources. Which rule uses the resources? rule build_datapackage takes resources (and some raw data) as input, as specified in the scenario_yaml. This makes it a bit more tricky.

jnnr commented 1 year ago

Found a solution for the above problem: Moving all rules that prepare resources in a submodule that can be included or not depending on a value in the settings. #290

jnnr commented 1 year ago

Almost there. Still missing:

jnnr commented 1 year ago

The resources can be downloaded, but there are differences to the ones created locally. First, the delimiter is different. Second, entries are in "". Added to the TODO list in first post.