Build subgraph for Residences = Households

[Richard-Deuchler]

I have finished a first version of the Residence analysis.

You can find the relevant CSV files (edges, nodes and shares) here:

Dropbox/Quintel/Projects/Restructure Research Dataset/Queries & CSV/CSVs/Residences

Note:

The "graph structure" needs to be adapted. It is decided that the two converters households_useful_demand_for_space_heating_after_insulation_for_houses_with_solar_heater and households_useful_demand_for_hot_water_for_houses_with_solar_heater. no longer have child_shares predefined in the graph structure Also see https://github.com/quintel/rdr/issues/34#issuecomment-21490631

There is one ‘extra’ CSV file in the dropbox directory, containing the production of solar PV in residences (in TJ). This needs to be added to the "central producers table": residences_solar_pv_solar_radiation_production_tj.csv

[antw]

@Richard-Deuchler I think I've done all I can with the residential data; here's the result:

• What can be calculated.
• What cannot be calculated.

I had to change queries to fix one or two incorrect CSV names, or add "quotes" around row or column names. I automated this with a script, and you can download my updated query CSVs from here.

The only problem with the share CSVs was that they referred to woodpellets, when it should instead be wood_pellets.

As for the calculated graph, there are a few issues to note:

• There is no demand for households_collective_government. This prevents that node, and also households_heat_network_connection_steam_hot_water from being calculated.
• Three CHP nodes receive energy from outside the residential sub-graph (the "super-source"), and do not have a demand defined. Other CHPs have their demands defined in the central producers CSV, but apparently not these three? The offending nodes are: households_space_heater_micro_chp_network_gas, households_water_heater_fuel_cell_chp_network_gas, and households_water_heater_micro_chp_network_gas.
• There exists the same issue with "savings" nodes on the left-hand side of the graph as in the building sector.

[ChaelKruip]

There is no demand for households_collective_government

I think @antw means households_collective_geothermal

Other CHPs have their demands defined in the central producers CSV, but apparently not these three?

The three CHPs mentioned are de-central producers and therefore not in the central producers CSV. Where they overlooked somehow? There may be something else confusing about them: they use both gas from final demand and from outside of the sector (through the super-sink in the current sub-graph). They are, however, not different from the rest of the heating options. They should be able to give a demand just in the same way as a combi boiler for example.

[antw]

I think @antw means households_collective_geothermal

Yes indeed. :blush:

The three CHPs mentioned are de-central producers and therefore not in the central producers CSV.

Fair enough, that makes sense. I see that all three of these CHP nodes get gas from energy_national_gas_network_natural_gas. If that node will have a demand, and the edges from the gas network to the CHPs have a share, then there's not really a problem here; it will resolve itself once we put all the sectors together. Is that the intention?

[antw]

If that node will have a demand, and the edges from the gas network to the CHPs have a share, then there's not really a problem here; it will resolve itself once we put all the sectors together. Is that the intention?

To answer my own question, I'm guessing this is not the intention. The energy queries on Dropbox don't do this, and ETEngine calculates these nodes from the outputs on the left. The demand for gas from these CHPs is then passed (to the right) to the gas network in the "energy" sector.

We could calculate these demands from the left, but Refinery cannot do that presently since those nodes have a undetermined demand also (households_locally_available_electricity, households_useful_demand_for_hot_water_after_solar_heater_and_add_on and households_useful_demand_for_space_heating_after_insulation_and_add_on).

[ChaelKruip]

@antw correct. @Richard-Deuchler and me will work on this in the afternoon. It will probably be easiest to specify the input share of network_gas (which part comes from final demand and which part comes from the central gas network) for the CHPs. Then Refinery can stay agnostic to the demand of the central gas network.

[Richard-Deuchler]

updates of analysis and Dropbox directory:

• added a query to nodes.csv that provides a demand to households_collective_geothermal
• added four child_shares that take care of the missing ....._savings_from_insulation converters.
• added three child_shares that take care of the missing natural gas that is demanded of energy_national_gas_network_natural_gas from the decentral residence CHPs

I hope that all converters can be now calculated . Maybe, you need to run that script in case the "quotation" marks got lost again.

[antw]

@Richard-Deuchler Thanks for the updated files. I'll add them to ETSource and test the results shortly. One thing I forgot to mention previously is the format of the share CSV files. The header names in these files should be "key" and "share".

Instead of:

residences_final_demand_coal_parent_share	nl
households_final_demand_for_space_heating_coal	1.0
households_final_demand_for_hot_water_coal	0.0

It should be:

key	share
households_final_demand_for_space_heating_coal	1.0
households_final_demand_for_hot_water_coal	0.0

I can make this change easily enough before adding them to ETSource, but would you be able to change whatever tool you use to export these files from the analyses? Or maybe there was a reason behind those header names that I'm not aware of?

[Richard-Deuchler]

right. I was not introduced to this new way of formatting. Can we agree on a compromise for now and use both headers? As in:

[antw]

right. I was not introduced to this new way of formatting. Can we agree on a compromise for now and use both headers? As in:

No, it has to be one or the other. The CSV format requires either: (1) no header row, (2) one header row. A second header row would be parsed as share data.

I'm unclear on why that naming format has been chosen, since the headers should identify what the column represents, i.e. the key of a "target" node, and the share of the edge. The headers currently appear to be a duplicate of the filename, and the region. The region can be fairly easily determined by the directory in which the CSV is stored:

• etsource/data/datasets/nl/shares

If there's reason behind the madness, I'll drop the matter and change our code to ignore the header row completely.

[ChaelKruip]

If there's reason behind the madness

I think I am to blame. I suggested to use the name of the worksheet (tab) as the name of the CSV file but Excel limits the naming of a sheet to 32 characters or something thereabouts. Therefore, to allow longer names and keep the possibility open to automate the generation of the CSV files (using VBA), I suggested to use the A1 cell of every sheet to store the name. Later the region was added to A2.

@antw If you feel that it is not useful to generate the CSVs using a (VBA) script or have another idea of how to stop this madness please let us know :wink:

[antw]

Here are the diagrams reflecting the recent changes:

• Initial values
• Calculable
• Incalculable

The addition of child_share values doesn't make any difference since they are typically used to calculate demand from left-to-right. If the left-hand node has undetermined demand, Refinery can't do anything with them.

For example:

Here child_share has been set on two household cooling edges; these tell Refinery that the nodes receive 100% of their "cooling" energy from the "after_insulation" parent node. Even though Refinery knows the demand of the "after_insulation" node, it doesn't have any information on how to split that demand between the "old" and "new" housing nodes.

Some of the shares are also inconsistent:

This is taken from the initial values diagram and shows that "final_demand_for_hot_water_network_gas" is to give zero energy to "fuel_cell_chp", while the other incoming fuel cell edge specifies that it gets only 75% of it's energy from the national gas network. So 25% of the input to the fuel cell is unaccounted for.

I added the fixed edge and node query CSVs to the "Residences" directory on Dropbox, so if further changes are required to the queries, you could use those files? Thanks!

P.S. If anything in the above diagrams isn't working, and you think that it should be, please let me know. I've been as thorough as I can, but time is not on my side. :wink:

[Richard-Deuchler]

Regarding the example:

@antw is of course very correct. I will do the math in Excel and provide a parent_share for households_useful_demand_for_cooling_after_insulation instead.

Regarding the inconsistency:

I meat to give input shares into the converters. For example: households_water_heater_fuel_cell_chp_network_gas is meant to receive 75% of its gas input from energy_national_gas_network_natural_gas (which happens to be 0 TJ) and 25% from households_final_demand_for_hot_water_network_gas (also 0 TJ).

I want to define input shares (child_shares) for three converters independently. Each of them receiving energy from two parents, which add up to 100%.

This is what I currently do:

I am a bit confused because I managed to provide child_shares in the context of the ..._from_insulation converters. Here, things seem to work differently. Would it be enough to change the from <> to in the residence_edges.csv ?

[antw]

Would it be enough to change the from <> to in the residence_edges.csv ?

In fact, the "from" and "to" nodes in the "residences_edges.csv" were in the wrong order and needed swapping, but I did that before producing the diagrams above:

I updated the file on Dropbox also.

I meat to give input shares into the converters. For example: households_water_heater_fuel_cell_chp_network_gas is meant to receive 75% of its gas input from energy_national_gas_network_natural_gas (which happens to be 0 TJ) and 25% from households_final_demand_for_hot_water_network_gas (also 0 TJ).

The edge from households_final_demand_for_hot_water_network_gas to households_water_heater_fuel_cell_chp_network_gas does have a query in "residences_edges.csv", but it specifies the parent share:

[Richard-Deuchler]

The edge from households_final_demand_for_hot_water_network_gas to households_water_heater_fuel_cell_chp_network_gas does have a query in "residences_edges.csv", but it specifies the parent share:

yes, correct. That parent_share is required. On top of it, I would like to specify that (from the perspective of the fuel_cell_chp) this energy input only makes up for 75%, while the remainder has to come from energy_national_gas_network_natural_gas. Let's have a little skype talk on that tomorrow, will make matters easier :cake:

[antw]

That parent_share is required. On top of it, I would like to specify that (from the perspective of the fuel_cell_chp) this energy input only makes up for 75%, while the remainder has to come from energy_national_gas_network_natural_gas.

Right, I understand; you want to set queries for both the parent share and child share on that edge:

• parent_share to specify that final_demand_hot_water... outputs 0% of it's energy through that edge;
• child_share to specify that fuel_cell_chp... receives 25% (or 75%) through that edge.

It's not possible right now to have more than one query per edge, but it will be very soon – probably by the end of the week, or early next.

Or perhaps, you want to specify the child share of one edge, and have Refinery automatically do the right thing with the other? Unfortunately, Refinery will not automatically figure out the other share, and you'll have to be explicit by specifying child shares for both the edges.

[wmeyers]

@antw, these multiple queries per edge/node are getting more and more needed huh? ;-) I'll talk to you about this on Skype.

@antw, @Richard-Deuchler, @ChaelKruip and @jorisberkhout: as far as I understood the csv files would be exported by VBA without the header specifying the full name. Only VBA needs it to know the name of the CSV and has to strip it. I think that would bring an end to the madness right? ;-)

[Richard-Deuchler]

I agree with @wmeyers on the VBA script. Would be great if we could have explicit naming in the Excel workbooks. These names would not be required if VBA automatically saves them to the right place.

[Richard-Deuchler]

putting this issue 'on hold' until multi-query support is enabled.

[jorisberkhout]

Agreed, I will make sure that the VBA script names the CSV files according to the header and strips the header line with the full name from the sheet before exporting it as CSV.

[ChaelKruip]

:+1:

[Richard-Deuchler]

One thing still is a bit unclear to me:

Parent_share In order to provide a parent_share, I write a CSV file that contains all shares (a,b,c,d), but I only write 3 lines in the edges.csv because refinery will figure out the 4th share (a+b+c+d=100%). Child_share Can I use the same procedure here? If p is known, will refinery derive q = 100% - p ? Or do I have to specifically tell refinery both shares in the edges.csv?

[Richard-Deuchler]

I also noticed that there are actually more occasions of _two queries per edge_:

• Two micro CHPs and the fuel cell (see above)
• The edges for ..._useful_demand_for_heating and ...._useful_demand_for_cooling will also have more than one query per edge. (located in the top left of the graph )

[Richard-Deuchler]

I am currently updating my analysis. Hope to be finished by Thursday afternoon.

Everything in Dropbox/Quintel/Projects/Restructure Research Dataset/Queries & CSV/CSVs/Residences will be retouched. It might be good to start from scratch again with the residence graph, so that we know that the Excel analysis actually produces all relevant data.

[antw]

If you know the child/parent share of all of the edges, please provide queries for them. You're right that in a lot of cases Refinery will be able to do the right thing, and figure out the share of the remaining edge; but not always.

The reason for this is that during Refinery's calculation phase it isn't actually calculating shares for edges, but demands. After it determines how much energy goes through an edge, it can normally determine the parent and child shares for that edge.

For example:

Here Refinery knows the parent share of three edges and the demand of the parent node. It can determine that 20 "energy" flows through a, b, and c. When it comes to d, it can figure out that 40 still needs to leave the node, and d is the only way for that to happen; therefore the demand of d is set to 40, and from that it determines that the parent share of d is 0.4.

However, take away the demand from the parent node, and Refinery cannot figure out the demands of a, b, and c. It also therefore can't determine the demand of d, or the parent share, even though you and I can see that it should be 0.4.

So... please be explicit and include queries for all shares that are specified in the share CSVs. Giving Refinery more information is always better than giving it less. :sunglasses:

[Richard-Deuchler]

aha :bulb: thanks!

[ChaelKruip]

even though you and I can see that it should be 0.4.

Note that 'we' make the implicit assumption here that parent shares should add up to 1. This is not the case for all nodes in the ETM unfortunately and it would therefore not be something to hard-code into Refinery IMO.

[Richard-Deuchler]

The file directory of the CSV files changed: Dropbox/Quintel/Projects/Restructure Research Dataset/analysis/analyses/6_residences/export

update: Please use the old directory for now. The new directory is not yet ready for use...

[Richard-Deuchler]

@antw, I have updated everything - it should work now. As soon as

• [x] two queries per edge can be handled by refinery,

you can give it another shot. :gun:

[Richard-Deuchler]

assigning to me removing 'on hold'

[Richard-Deuchler]

Households should fall into place when https://github.com/quintel/refinery/issues/37 is closed.

[Richard-Deuchler]

closing.