Making the load profiles etc. independent of the national dataset

[grdw]

Currently the local dataset "inherits" all the csv files from the national dataset. This should be the case? Or not be the case.

It should at least be possible to overwrite them. I think that this can work as:

datasets/ameland
datasets/nl
├── load_profiles
│   └── wind_offshore.csv

In this case the wind_offshore profile of nl is used.

datasets/ameland
├── load_profiles
│   └── wind_offshore.csv
datasets/nl
├── load_profiles
│   └── wind_offshore.csv

In this case the wind_offshore of ameland is used.

[grdw]

Yesterday during our (@ploh and me) standup there was some discussion about this in relation to FCE. A dataset doesn't need to have FCE attributes in order to be properly calculated and used in ETEngine for a scenario. So, when not having FCE related *.yml files in the fce folder, should you still inherit them from the base dataset for a local dataset or just ignore them fully.

I think that this should be set with the has_fce flag in the dataset.

What do you think? Including @AlexanderWirtz @DorinevanderVlies @ploh

[ploh]

I would propose the following solution:

• When creating a derived dataset via rake scale in ETSource, the load_profiles, fce and network folders will be copied from the base dataset to the derived dataset.
• If you do not want to use fce (resp. network) (load_profiles are mandatory) at all, you can delete the folder and set the corresponding attribute in the derived dataset ad file (has_fce (resp. use_network_calculations)) to false. This works just like for normal full datasets, nothing special here.
• If you want to use the base dataset instead - and thereby have updates to the base dataset's files propagated to the derived dataset - you can set the (to be invented) optional attribute fallback_to_base_dataset.(fce | load_profiles | network) in the derived dataset ad to true (and also delete the folder in the derived dataset).

This way we make it possible to share data from the base dataset but do that only very explicitly - so that (a) we do not share by accident and (b) it is still possible to not use fce / network at all.

What do you think? @grdw @antw @ChaelKruip @AlexanderWirtz @DorinevanderVlies @jorisberkhout

Also, setting has_fce to false is different from deleting the fce folder: If the fce folder exists, it is still partially used even if has_fce is disabled (Only the :co2_conversion_per_mj attribute is used but the Co2 is still computed according to country-of-origin shares). On the other hand, it is possible to activate FCE in a scenario if has_fce is set even if no fce folder exists - which is probably a bug (Atlas should probably reject a dataset with has_fce but no fce folder. What do you think? @ChaelKruip).

[ploh]

@antw My code search in ETEngine found that fce, network and load_profiles (and of course time_curves which even need to be scaled) are used at runtime but demands, efficiencies, shares, central_producers (in the local-datasets branch), primary_production and energy_balance are probably not, i.e. they are only used in Atlas / Refinery when building the graph. Does that sound plausible to you?

[antw]

• If you want to use the base dataset instead - and thereby have updates to the base dataset's files propagated to the derived dataset - you can set the (to be invented) optional attribute fallback_to_base_dataset.(fce | load_profiles | network) in the derived dataset ad to true (and also delete the folder in the derived dataset).

I like the rest of the proposal, but I'm not sure this is necessary. If someone really wants to use the original FCE, network, or load profile data: use a symlink! You'd get all the same functionality without having to write any extra code. :wink:

i.e.

pwd
# etsource/datasets/limburg

ln -s ../../nl/fce

ls -l
# ...
# lrwxr-xr-x  1 user  group  9 11 Jan 13:03 fce -> ../nl/fce
# ...

My code search in ETEngine found that [...] are used at runtime but demands, efficiencies, shares, central_producers, primary_production and energy_balance are probably not

That sounds about right. :+1:

[ploh]

If someone really wants to use the original FCE, network, or load profile data: use a symlink! You'd get all the same functionality without having to write any extra code. :wink:

:+1: I am just not sure if symlinks in Git repositories will work under Windows... but fortunately we are not using that anyways :stuck_out_tongue_winking_eye:

[grdw]

What is the current status of this issue? @ploh

[ploh]

What is the current status of this issue? @ploh

I have done some research and I will implement it when I am done with the scaling. Will probably only take a couple hours, max.