ESDL to JSON to Julia graph structure

[suvayu]

• [x] Convert ESDL to JSON using esdl2json (based on code by @Lokkij)
• [x] Create the graph edges/flows
  • for posterity: TulipaEnergy/TulipaEnergyModel.jl#359
  • for current implementation, see: src/parsers.jl
• [ ] Read other flow related attributes; ~see FlowData~
• [ ] Read flow profiles; ~see FlowProfiles~
  • [x] TulipaEnergy/TulipaIO.jl#6 Every ReferenceProfile in an ESDL file has to be retrieved from InfluxDB. There's no Julia client, so we wrote one.
• [ ] Combine the flow definitions with the profiles
  • The data structure that is used by the JuMP model is GraphFlowData, which combines (3-4).
• [ ] Read assets with its attributes; see AssetData, GraphAssetData

[clizbe]

Related to: TulipaEnergy/TulipaEnergyModel.jl#89 TulipaEnergy/TulipaEnergyModel.jl#289

[datejada]

Comments from the workday meeting:

assets_data.csv

• consumers:

Tulipa	ESDL
active	state
peak_demand	power

• producers:

Tulipa	ESDL	Comment
active	state
investable		Ask to add it in ESDL
investment_cost	asset.CostInformation.investmentCosts	the unit doesn't match
variable_cost	asset.CostInformation.variableOperationalAndMaintenanceCosts	If there is no data, we put zero for the time being
lifetime	TechnicalLifetime
investment_limit	Potential?	We need to include it in ESDL or try to use Potential property of the area
capacity	power
initial_capacity		There is no data in ESDL; ask to add it

• conversion: same as producers, but the efficiency from the asset goes to the outgoing flow. For assets connected to two carriers, then, we need two different efficiencies between the input and outputs; ESDL has electrical efficiency and heat efficiency (but not for all assets; we need to double-check) For heat pumps -> COP to efficiency and other conventions assets -> Efficiency to efficiency

• storage: same as producer, but we need more parameters

Tulipa	ESDL	Comment
capacity	maxDischargeRate and maxChargeRate	Tulipa uses only one value, but might change it to consider both
initial_storage_level	fillLevel	ESDL uses p.u., Tulipa uses MW
energy_to_power_ratio	?	Include in ESDL? or calculate it from the ratio of capacity (MWh)/maxDischargeRate (MW)
initial_storage_capacity	capacity	double-check the units ESDL uses J and we use MWh

The charging and discharging efficiencies are defined in the asset but must be mapped to the flow asset data.

flows_data.csv

is_transport: false, then get the efficiencies from the asset is_transport: true is similar to producers, but we need to create two new parameters: initial_import_capacity and initial_export_capacity instead of only the initial_capacity

[datejada]

@wcoenraads, we need to add data to the Norse Case Study from a random sample of one of our TNO databases. Could you help us with that?

[wcoenraads]

Sure! We need two things for this:

• A ReferenceProfile in the ESDL that references the profile
• Functionality to use the information in the ReferenceProfile to actually retrieve the data from the database

The first is pretty straightforward (I can provide examples). The second is also reasonably easy - if the database is InfluxDB I have some existing code we could use for this. Do you have specific data (or a specific database) in mind?

[suvayu]

Looks like there is no reliable Julia client for InfluxDB. @wcoenraads, I presume your existing code is in Python?

My questions:

• If there is no client library, is the recommended way to connect using REST API?
• I'm guessing we would also need to manage a "secret token"?

[wcoenraads]

Oh true, it's Python. Shame that there's no library for Julia, in that case we'll indeed have to use the REST API. I have no experience using it, but it should be straightforward enough.

With regards to authentication, I believe many TNO databases (like the EDR) don't require a token - they are freely accessible from the TNO network. But that depends on the specific database you want to pull data from, and users outside of TNO would not be able to access those DBs anyway so we'll need to think about the situation there.

[suvayu]

Hmm, if the DBs are internal only, we need to find a way for eSci people to work with it during the development phase at least (VPN?).

@datejada @clizbe could you have a look? Either choose a DB without this restriction, or figure out if there's some paperwork to be done.

[suvayu]

Oh true, it's Python. Shame that there's no library for Julia, in that case we'll indeed have to use the REST API. I have no experience using it, but it should be straightforward enough.

I created an issue and assigned it to you. Please let me know if something is unclear.

[lsoucasse]

@suvayu: Attached an esdl file with some data for the tiny test case. This was created some time ago so I might have chosen some fields that are not in agreement with what we discussed last week. TinyTest.txt

[clizbe]

@lsoucasse Thanks! But this doesn't cover all the features in the table above. @wcoenraads I'll try making one, but can you also help with this?

[clizbe]

For "investable" - maybe that shouldn't be in ESDL, but rather a scenario parameter?

[clizbe]

Using this to track my progress updating the ESDL file:

Tulipa	ESDL	Comment	Added to Norse ESDL
CONSUMERS
active	state (ENABLED/DISABLED)	X
peak_demand	power	X
:---------------	:----------	:----------	:------
PRODUCERS
active	state		X
investable		Not in ESDL - add or have separate file?
investment_cost (kEUR/MW/year)	asset.CostInformation.investmentCosts (EUR/MWh/yr)	the unit doesn't match	X
variable_cost	asset.CostInformation.variableOperationalAndMaintenanceCosts	If there is no data, we put zero for the time being	X
lifetime	TechnicalLifetime		X
investment_limit	Potential?	Not in ESDL - add or have separate file or try to use Potential property of the area
capacity	Not in ESDL - ask to add	This is the upgrade increment when invested in
initial_capacity	power		X
:---------------	:----------	:----------	:------
CONVERSION
active	state		X
investment_cost (kEUR/MW/year)	asset.CostInformation.investmentCosts (EUR/MWh/yr)	the unit doesn't match	X
variable_cost	asset.CostInformation.variableOperationalAndMaintenanceCosts	If there is no data, we put zero for the time being	X
lifetime	TechnicalLifetime		X
investment_limit	Potential?	We need to include it in ESDL or try to use Potential property of the area
capacity	Not in ESDL - ask to add	This is the upgrade increment when invested in
initial_capacity	power		X
efficiency	efficiency or InputOutputRelation	* See below
:---------------	:----------	:----------	:------
STORAGE
active	state		X
investment_cost	asset.CostInformation.investmentCosts	the unit doesn't match	X
variable_cost	asset.CostInformation.variableOperationalAndMaintenanceCosts	If there is no data, we put zero for the time being	X
lifetime	TechnicalLifetime		X
investment_limit	Potential?	We need to include it in ESDL or try to use Potential property of the area
capacity	Not in ESDL - ask to add	This is the upgrade increment when invested in
capacity (charge/discharge)	maxDischargeRate and maxChargeRate	Tulipa uses only one value, but might change it to consider both **	X
initial_storage_level	fillLevel	ESDL uses p.u., Tulipa uses MW	X
energy_to_power_ratio	?	Include in ESDL? or calculate it from the ratio of capacity (MWh)/maxDischargeRate (MW)
initial_storage_capacity	capacity	double-check the units ESDL uses J and we use MWh	X

*The efficiency from the asset goes to the outgoing flow. For assets connected to two carriers, then, we need two different efficiencies between the input and outputs; ESDL has electrical efficiency and heat efficiency (but not for all assets; we need to double-check), For heat pumps -> COP to efficiency and other conventions assets -> Efficiency to efficiency ** The charging and discharging efficiencies are defined in the asset but must be mapped to the flow asset data.

We can do multiple efficiencies using this InputOutputRelation feature. It doesn't exist yet in the MapEditor, but it's in ESDL and we can push for it to be added to the MapEditor:

[clizbe]

Things ESDL appears to be missing. We can either ask them to add these, or they should be in a different file:

• investable - whether or not an asset can be invested in
• investment_limit - max you can invest in an asset (or maybe potential?)
• capacity - this should be renamed as it's confusing with initial_capacity - this "capacity" is the upgrade increment when invested in
• efficiency (multiple) - exists in ESDL as InputOutputRelation, but not yet in MapEditor
• Also: Ask about changing "Power" to "Capacity" in ESDL.

[clizbe]

@suvayu Here's the new ESDL with as many features as I could include (see table above).

Norse Mythology.txt

[suvayu]

@datejada @clizbe I found an instance where merging the attributes from the "from" and "to" nodes has conflicts (see the norse mythology file, also in the repo):

ERROR: DomainError with Following fields have conflicting values:
┌──────────────────┬────────────┬───────────────┐
│           fields │ Waste heat │ HeatPump_4b33 │
├──────────────────┼────────────┼───────────────┤
│ initial_capacity │      100.0 │          10.0 │
│  investment_cost │       14.0 │          16.0 │
│    variable_cost │        2.0 │          15.0 │
└──────────────────┴────────────┴───────────────┘

Is this realistic? Any thoughts how to resolve this kind of conflicts?

[suvayu]

Type	Tulipa	ESDL
Storage	capacity (charge/discharge)	maxDischargeRate and maxChargeRate

@clizbe I'm not sure I understand this entry from the table. ESDL has two different rates, whereas Tulipa has one. How does that map?

[clizbe]

That's up for debate. We'll most likely change Tulipa to 2.

[clizbe]

@datejada @clizbe I found an instance where merging the attributes from the "from" and "to" nodes has conflicts (see the norse mythology file, also in the repo):

ERROR: DomainError with Following fields have conflicting values:
┌──────────────────┬────────────┬───────────────┐
│           fields │ Waste heat │ HeatPump_4b33 │
├──────────────────┼────────────┼───────────────┤
│ initial_capacity │      100.0 │          10.0 │
│  investment_cost │       14.0 │          16.0 │
│    variable_cost │        2.0 │          15.0 │
└──────────────────┴────────────┴───────────────┘

Is this realistic? Any thoughts how to resolve this kind of conflicts?

Ah this is just because I filled in random numbers. You guys said that was fine! Is it just a value conflict that doesn't make sense?

[suvayu]

Is it just a value conflict that doesn't make sense?

Yes. Basically the logic is:

• if for asset 1 it's missing and asset 2 is present, accept value set with asset 2
• if asset 1 and asset 2 has set the same value, accept it
• if both assets set the value but thwy aren't equal, error

This is mostly to be conservative yet reasonable. If the rubric above looks good, I'll just make the esdl file conformant and proceed.

[suvayu]

We'll most likely change Tulipa to 2.

In that case, for now I'll just pick one, and implement the other one but leave it commented out.

[clizbe]

Last step is testing with existing input flow.

[abelsiqueira]

Some comments on the ESDL/Tulipa compatibility were added here: https://github.com/TulipaEnergy/TulipaEnergyModel.jl/discussions/606