First run of GER 2011 graph

[Richard-Deuchler]

I would like to test the German scenario that I have built. I have created

• a corrected energy balance
• CSV files
• primary production and central producers tables

I will copy those to etsource into a similar folder structure that we have for NL.

How do I get atlas to run on the german data (locally)? I need to produce a German static.yml.

[Richard-Deuchler]

@antw, could you give me an estimate on when we can tackle this issue? I would very much like to have a German scenario running locally before the end of this week. We want Germany to be part of the September deploy

[antw]

The debug:graph script is hard-coded to use the NL region, but it should be trivial to provide an option to change the country being calculated. I'll try to make the necessary changes today.

[Richard-Deuchler]

:+1:

[antw]

@Richard-Deuchler The Atlas debug:graph and debug:check tasks will now take a "DATASET" argument, so that you can choose which one you want to calculate. After updating Atlas, it may be best to also run bundle install to get the latest Refinery as it had some small improvements recently.

bundle exec rake debug:graph              # Defaults to "nl"
bundle exec rake debug:graph DATASET=nl   # Set to "nl"
bundle exec rake debug:graph DATASET=de   # Set to "de"

Since you mentioned static.yml specifically: You won't be able to use this in ETEngine just yet; do you also need that by the end of the week? It should be no problem to have it ready by then, but if you need it sooner (e.g. tomorrow) please let me know!

[Richard-Deuchler]

Thanks. Calculating Germany works, but refinery does not produce a complete graph yet (9 nodes missing). The problem is caused by hard-coding some import/export converters to 0.0 (this was tailored to fit NL, but GER does not export gas....)

We are working on it.

[Richard-Deuchler]

The problem is a bit larger than we hoped:

1. We either need to set import or export of natural_gas to 0.0. Since NL is a net exporter of gas, we had set the export converter to 0.0. For Germany, we need to do the exact opposite, because it is a net importer....
2. Since Germany does not demand any biogas in the sectors, the converter energy_distribution_biogas cannot be calculated (child shares are useless if they are all equal to 0).

Regarding the static.yml: It would be great if I could look at a German et-model by the end of tomorrow. Maybe with one of the following:

• an ugly hack that works on my machine (pretending that it is nl data :smile: )
• a static.yml that works on my local machine
• or directly on staging. Germany already shows up on the dropdown menu, but charts and sliders do not display ( I guess some area codes are hard-coded to nl.) Does staging require a static.yml at this point?

Lets keep in touch tomorrow.

[Richard-Deuchler]

The problem with the biogas probably originates from the Excel analyses. I hope we can solve it there. The problem that refinery has with determining whether a country is a net a importer/exporter needs to be addressed in a modelling meeting.

[StijnDellaert]

You're right Richard. On problem 2: In the primary production analysis we actually calculate the amount of biogas that is needed, but we don't query the production directly to the energy_production_biogas converter. This was overlooked since for the Netherlands the production can be calculated using a share of energy_upgrade_biogas of energy_distribution_biogas.

I think we can solve this part of the problem rather easily by adding energy_production_biogas to the exported production csv. This would also mean that we have to remove the production of manure and corn from the analysis since they are solely used to produce biogas and will follow automatically

[wouterterlouw]

I think we can solve this part of the problem rather easily by adding energy_production_biogas to the exported production csv.

Be aware that as result of setting the energy_production_biogas and the share of energy_upgrade_biogas validation errors can occur as result of incorrect rounding.

[antw]

@Richard-Deuchler I have updated ETEngine to support the German dataset. There are instructions for what you need to do in the commit message, but if you have any questions or difficulties, please don't hesitate to ask. You will need to fetch the latest ETEngine (master branch) and then bundle install to ensure that it uses the newest version of Atlas.

I have uploaded new YML files for both NL and DE. Your ETEngine directory should end up looking something like this:

I tried it myself, and found that there are some discrepancies between the values calculated in Refinery, and those calculated in ETEngine:

[antw]

BTW, I uploaded the validation Excel file to Dropbox if you want to take a look yourself.

[wouterterlouw]

I tried it myself, and found that there are some discrepancies between the values calculated in Refinery, and those calculated in ETEngine

@Richard-Deuchler This is caused by the problems in the Metal sub-sector. The shares for the energy_distribution_coal_gas converter to initialize the transformation part of the metal sub-sector are calculated based on newly calculated efficiencies in the analysis, while ETengine still runs on the efficiencies that were calculated for the Netherlands.

See https://github.com/quintel/rdr/issues/107 and the second paragraph of https://github.com/quintel/rdr/issues/87 for some background information.

[Richard-Deuchler]

@wouterterlouw, thanks for the comment! Let's solve the problem with the country specific meta_efficiencies in a new ticket.

@antw, looks like great work! I think we can consider the "first run of a German scenario" successful and close this ticket.

One thing still remains unclear to me: Which static.yml does beta.pro.et-model.com use? Where does it live and how is it updated when we change something on etsource?

[antw]

One thing still remains unclear to me: Which static.yml does beta.pro.et-model.com use? Where does it live and how is it updated when we change something on etsource?

Beta is using an NL static.yml I created yesterday (when it was deployed). Unfortunately I have to create and upload these manually; the plan is to have ETEngine automatically create the YML files itself when deployed (or when a change to ETSource is detected) but that isn't implemented yet. Hopefully by the end of the week...