search

spine-tools / SpineOpt.jl

A highly adaptable modelling framework for multi-energy systems
https://www.tools-for-energy-system-modelling.org/
GNU Lesser General Public License v3.0
49 stars 12 forks source link

Investigate and improve constraint_connection_flow_capacity_indices #925

Open abelsiqueira opened 4 months ago

abelsiqueira commented 4 months ago

@suvayu and I ran the example included in #896 with log_level=3 and the largest memory hog is constraint_connection_flow_capacity. Looking into the profile, the function constraint_connection_flow_capacity_indices seems to be one of the main culprits - which aligns with what @manuelma and @Tasqu found out some time ago.

Running

m = run_spineopt(url_in, url_out)
@time v = SpineOpt.constraint_connection_flow_capacity_indices(m) |> collect

reports a 4.431 GB memory use for a sizeof(v) = 17385984 (~17 MB) object, so we think this and other similar functions are a good place to start exploring and improving.

@manuelma, @Tasqu, are you working on this? How can we coordinate this to avoid duplicating work?

Tasqu commented 3 months ago

@abelsiqueira at the moment, I'm not working on actively on SpineOpt at all. I'm busy with other project work in the foreseeable future.

manuelma commented 3 months ago

@abelsiqueira I think it's safe to go ahead, no need for coordination from my part at this point. Good luck!