search

fslaborg / flips

Fsharp LInear Programming System
https://flipslibrary.com/#/
MIT License
251 stars 32 forks source link

Out of memory only for CBC and GLOP #179

Closed sloaneross closed 2 years ago

sloaneross commented 2 years ago

When running a particularly large problem I get a "Out of Memory" output when ram use hits 10GB. On Mac I get:

System.OverflowException: Arithmetic operation resulted in an overflow.
   at System.StubHelpers.AnsiCharMarshaler.DoAnsiConversion(String str, Boolean fBestFit, Boolean fThrowOnUnmappableChar, Int32& cbLength)

This only happens on CBC or GLOP, Gurobi works just fine, which makes me think it's something in the OR tools setup. No idea where to start debugging though.

matthewcrews commented 2 years ago

Could you give me a way to recreate the issue?

How many Constraints and Decision Variables are in your model? Also, are you using any non-ANSI compatible characters in your naming? I ask because of the error message you posted.

sloaneross commented 2 years ago

I'm not sure I can provide a way to recreate the issue, what I can say is we are running this on a very large model and giving each decision a descriptive (i.e. long) name. I'm going to try shortening the names and see if that helps.

We have close to 800,000,000 variables total and we do not get this error when we run our model in a smaller timeframe. This may be the ticket!

matthewcrews commented 2 years ago

Wow, that's a big problem. Let me know if the name shortening resolves the issue.

sloaneross commented 2 years ago

I shortened our decision names to the string representation of an index + variable type (we have 8 types of variables), to try to minimize the string length while keeping everything unique. This yielded the same error with our 'small' problem ( > 142,000,000 variables).

-- this gives the same ANSI error as before at ~ 8GB of ram use. Using DotMemory I can see that the majority of the memory is coming from unmanaged sources during the for loop of addConstraints. If that helps any.

sloaneross commented 2 years ago

I made a new version of Flips for myself using the unnamed version of 'MakeConstraint' and I get an out of memory error still, so I think we can conclusively scratch that off.

matthewcrews commented 2 years ago

Is there any way you can get me an example problem that reproduces this behavior? It won't be possible to diagnose otherwise. I'm betting this is a problem with Google OR Tools and not Flips since Gurobi works.

matthewcrews commented 2 years ago

I'm going to close this since I do not believe this is a problem with Flips but with the OR Tools library. I recommend raising an issue with them. If you are able to provide some code which reproduces the error, I would be happy to look into it further.