Move LOLE calculation to Merit

[ChaelKruip]

The LOLE calculation on ETE currently compares the reliable installed capacity to the demand curve for every hour and counts the hours that demand exceeds capacity. This calculation can be incorporated into Merit quite naturally as all the required information is known to Merit.

It would be best if the LOLE functionality could be called independently from the merit order calculation though for reasons of performance and clarity.

[jorisberkhout]

YES! I am very much in favour of doing this.

In addition to making the calculation more transparent, it also reduces the research effort from trying to find the capacity_credit to finding reliable load_profiles (which we already have in many cases).

[antw]

I started working on this, but I'm no longer convinced it's the right thing to do:

• The demands used to determine LOLE all originate in ETEngine, not the Merit order – whether it be a custom total demand query, a custom production query, or demands for wind/solar.
• The Merit gem is given the load profiles by ETEngine; it does not know how to load the curves itself – ETEngine has Atlas load them via the MeritOrderInjector.

My preference would be to keep the LOLE calculation in ETEngine, but move it out of GraphApi into a dedicated LOLE calculation module. Otherwise, we'll move the core feature to Merit but still need just as much code in ETEngine to set it up.

However, my argument is a little less compelling if the feature is going to be needed outside of ETEngine (e.g. merit-convergence, DECC project, etc).

[antw]

Discussed with @ChaelKruip, and this will be added to Merit after all. The feature may be desirable for the DECC project, and must-run data is already available in the M/O without having to actually run the calculation.

[dennisquintel]

I think the general challenge here is that we have different 'applications'/'modules' nowadays, and they all need to communicate with each other, E.g:

• Merit needs to have a 'total demand', which can come from etengine, or from the DECC model.
• Etengine needs to get the FLH's, which can come from ETSource (dictated by the dataset) or by Merit
• ETLoader needs to know the electricity demand in households, or the number of electric cars in a certain area, which can be obtained from etengine/etsource

Before, we could just send the required data over, and that worked just fine, but I think it is time now to think about a more structural approach to this.... what do you think @ChaelKruip and @antw?

[ChaelKruip]

I think the general challenge here is that we have different 'applications'/'modules' nowadays, and they all need to communicate with each other.

I fully agree. I think we need to ensure that communications happen in a uniform way for all modules (and the ETM).

• Data fed to Merit (for instance) should be in the same format whether it comes from a 'stub' or ETE or another module.
• Reversely, an amount of energy should also have the same fixed unit and precision whether it comes out of Merit or any other module.
• Perhaps we also should think about a more rigorous way of defining order of execution / communication of the various modules. It might make a lot of difference if we run ETLoader first and Merit second or the other way around for the result of a calculation.

[antw]

Before, we could just send the required data over, and that worked just fine, but I think it is time now to think about a more structural approach to this....

In time, I'd very much like to be rid of the "monolithic" ETEngine; we've talked in the past about turning ETEngine into a RubyGem, but I think I'd like to go further and extract major features to entirely separate processes. This gives us (1) strong separation of concerns, (2) easier testing, (3) the possibility for each component to use different (more appropriate) languages without having to rewrite everything.

For example:

Alternatively, Celluloid may provide the means to do it all in one big process, but I don't know much about that at the moment.

I think this dream scenario is some way off – and I certainly don't have all the answers yet – but hopefully ETLoader will be the first step towards it. :sunglasses:

[ChaelKruip]

For example:

I like the direction this is going! One question though: why would ETSource coincide with the role of 'coordinator'? Is there a specific reason not to split off ETSource as a separate module as well and have a dedicated 'coordinator' handle the interfacing between everything?

[antw]

One question though: why would ETSource coincide with the role of 'coordinator'? Is there a specific reason not to split off ETSource as a separate module as well and have a dedicated 'coordinator' handle the interfacing between everything?

The diagram doesn't make this clear, but the "co-ordinator" and "ETSource" remain separate projects (or "modules", "repositories", whatever you want to call them). However, ETSource doesn't need to be a separate operating system process. I see the co-ordinator as loading the ETSource data, and providing sub-sets of that data to the other processes:

[ChaelKruip]

I see the co-ordinator as loading the ETSource data, and providing sub-sets of that data to the other processes

That makes a lot of sense to me now :smile:

[dennisquintel]

Excellent work and write up, @antw. I especially like that little crown on top of the coordinator. :smile:

One question though:

Would it be possible for the coordinator to get information from other sources than ETSource? This is required when we want to run Merit for other models (e.g. DECC).

[antw]

Would it be possible for the coordinator to get information from other sources than ETSource?

I haven't really thought any of this through properly, it was really just a brain-dump of how I think things could be organised eventually. What that diagram depicts is an ETEngine replacement.

If all of the processes have clearly-defined public interfaces, we could have a separate DECC web API (Sinatra, Graph, whatever) which talks directly to the Merit process.

[dennisquintel]

Question that arrises with me: would it be the responsibility of the coordinator to coordinate requests from an external app/vendor to just one of the projects/modules/services, or would they be able to talk to one directly?

[antw]

would it be the responsibility of the coordinator to coordinate requests from an external app/vendor to just one of the projects/modules/services, or would they be able to talk to one directly?

I don't know. I have literally put as much thought into this as deciding what meal to eat tonight. :laughing:

I would not expect each process to have it's own web API, so either we'd write them as-and-when other projects needed them, or we'd have a way to tell the co-ordinator which components we want to use. The idea is that the "web API" in the above diagram is a replacement for the Rails part of ETEngine, and the co-ordinator is new, and manages calculating the graph, deciding which queries are to be run, etc.

Anyway, back to the real issue:

@ChaelKruip One minor problem I've hit is that the current LOLE calculation and peak electricity demand query can be run on both the present and future graphs, but Merit is only set up on the future graph.

I think this means we're going to have to also set up (but not run) the M/O for the present graph? I'm guessing we can't do away with the present values for these two queries?

[ChaelKruip]

One minor problem I've hit is that the current LOLE calculation and peak electricity demand query can be run on both the present and future graphs

I discussed this with @jorisberkhout and we think that only the future value is relevant for now. I suggest to keep the 'future only' loader intact for the time being. I'll think about possible applications / uses where we might be needing Merit / LOLE to run on both present and future graphs but right now I don't think we need it.

[ChaelKruip]

Closing.