Closed grdw closed 7 years ago
Yesterday during our (@ploh and me) standup there was some discussion about this in relation to FCE. A dataset doesn't need to have FCE attributes in order to be properly calculated and used in ETEngine for a scenario. So, when not having FCE related *.yml
files in the fce
folder, should you still inherit them from the base dataset for a local dataset or just ignore them fully.
I think that this should be set with the has_fce
flag in the dataset
.
What do you think? Including @AlexanderWirtz @DorinevanderVlies @ploh
I would propose the following solution:
rake scale
in ETSource, the load_profiles
, fce
and network
folders will be copied from the base dataset to the derived dataset.fce
(resp. network
) (load_profiles
are mandatory) at all, you can delete the folder and set the corresponding attribute in the derived dataset ad
file (has_fce
(resp. use_network_calculations
)) to false
. This works just like for normal full datasets, nothing special here.fallback_to_base_dataset.(fce | load_profiles | network)
in the derived dataset ad
to true
(and also delete the folder in the derived dataset).This way we make it possible to share data from the base dataset but do that only very explicitly - so that
(a) we do not share by accident and
(b) it is still possible to not use fce
/ network
at all.
What do you think? @grdw @antw @ChaelKruip @AlexanderWirtz @DorinevanderVlies @jorisberkhout
Also, setting has_fce
to false
is different from deleting the fce
folder:
If the fce
folder exists, it is still partially used even if has_fce
is disabled (Only the :co2_conversion_per_mj
attribute is used but the Co2 is still computed according to country-of-origin shares).
On the other hand, it is possible to activate FCE in a scenario if has_fce
is set even if no fce
folder exists - which is probably a bug (Atlas should probably reject a dataset with has_fce
but no fce
folder. What do you think? @ChaelKruip).
@antw My code search in ETEngine found that fce
, network
and load_profiles
(and of course time_curves
which even need to be scaled) are used at runtime but demands
, efficiencies
, shares
, central_producers
(in the local-datasets branch), primary_production
and energy_balance
are probably not, i.e. they are only used in Atlas / Refinery when building the graph. Does that sound plausible to you?
- If you want to use the base dataset instead - and thereby have updates to the base dataset's files propagated to the derived dataset - you can set the (to be invented) optional attribute fallback_to_base_dataset.(fce | load_profiles | network) in the derived dataset ad to true (and also delete the folder in the derived dataset).
I like the rest of the proposal, but I'm not sure this is necessary. If someone really wants to use the original FCE, network, or load profile data: use a symlink! You'd get all the same functionality without having to write any extra code. :wink:
i.e.
pwd
# etsource/datasets/limburg
ln -s ../../nl/fce
ls -l
# ...
# lrwxr-xr-x 1 user group 9 11 Jan 13:03 fce -> ../nl/fce
# ...
My code search in ETEngine found that [...] are used at runtime but
demands
,efficiencies
,shares
,central_producers
,primary_production
andenergy_balance
are probably not
That sounds about right. :+1:
If someone really wants to use the original FCE, network, or load profile data: use a symlink! You'd get all the same functionality without having to write any extra code. :wink:
:+1: I am just not sure if symlinks in Git repositories will work under Windows... but fortunately we are not using that anyways :stuck_out_tongue_winking_eye:
What is the current status of this issue? @ploh
What is the current status of this issue? @ploh
I have done some research and I will implement it when I am done with the scaling. Will probably only take a couple hours, max.
Currently the local dataset "inherits" all the csv files from the national dataset. This should be the case? Or not be the case.
It should at least be possible to overwrite them. I think that this can work as:
In this case the
wind_offshore
profile ofnl
is used.In this case the
wind_offshore
ofameland
is used.