Extremely high memory usage (114 GB for a 8000 bus case) and unable to solve a single snapshot ED problem (no contingency)

[mzy2240]

Description

A clear and concise description of what the bug is. An incredible 114GB of memory usage observed during the optimization process, and failed in the third iteration.

Steps to Reproduce

example2.txt Rename to json file and load the instance to reproduce. Use HiGHS as the optimizer.

Please describe how can the developers reproduce the problem in their own computers. Code snippets and sample input files are specially helpful. For example:

1. Install the package
2. Run the code below with the attached input file...
3. The following error appears...

System Information

• Operating System: [e.g. Ubuntu 20.04] Windows 10
• Julia version: [e.g. 1.4] 1.8.5
• Package version: [e.g. 0.0.1] 0.3.0

[mzy2240]

I fixed the offer curve in the example. Considering it is just an economic dispatch problem (single snapshot, no contingency, so not SCED), it should be very efficient to solve, and should not use 114 GB to handle a <9,000 bus case.

[mzy2240]

The extreme memory usage might be a memory leak, and seems to come from this code:

time_screening = @elapsed begin
        violations = []
        for sc in model[:instance].scenarios
            push!(
                violations,
                _find_violations(
                    model,
                    sc,
                    max_per_line = method.max_violations_per_line,
                    max_per_period = method.max_violations_per_period,
                ),
            )
        end
    end

The JuMP model itself does not introduce new variables or constraints in the above code in my example, so really no reason for such a huge memory usage (and more importantly, not GC-able).

[mzy2240]

Ok I am able to solve the problem by removing the custom offset for buses. So I think that's the reason. I am wondering if the default solution method has to work with the hardcoded bus offsets @iSoron

[iSoron]

@mzy2240 I'm still investigating this issue, but the package should maintain the bus sequence from the input file. If, for any reason, you would like a certain bus to be the slack, you can list it first.

[iSoron]

@mzy2240 For this bug report, could you please confirm that no changes have been made to the source code of the package? Are you experiencing this behavior on a clean copy of the dev branch?

[mzy2240]

Yes still experience the same issue in a clean copy of the dev branch without any changes on the source code. The Windows machine is a dual socket workstation just fyi. Maybe it has something to do with the dual socket setup? The Julia version is 1.9.2.

[mzy2240]

Should be a false alarm caused by local change.