Out of memory - Githubissues

IBMDecisionOptimization / docplex-examples

These samples demonstrate how to use the DOcplex library to model and solve optimization problems.

https://ibmdecisionoptimization.github.io/

Apache License 2.0

396 stars 228 forks source link

Out of memory #37

Closed Milad321 closed 3 years ago

Milad321 commented 3 years ago

Hello, I am using the docplex for my project. In our model there are some constraints which when we increase the number of the samples (that will lead to the number of constraints to be increased), CPLEX Error 1001 (out of memory) occurs and it shows this Message: " cplex.exceptions.errors.CplexSolverError: CPLEX Error 1001: Out of memory" when we building the model. However the RAM Utilization is 50% and 16GB of 32GB RAM is free. OS= Windows 10 64bit Python version= 3.7 docplex version=2.18.200 CPLEX version= enterprise server 12.10.0

I would be very grateful if you help me how I can solve this problem.

vlkong commented 3 years ago

@Milad321 Hi Milad. Can you provide use the model ? I understand that because input data is from your simulation environment, we might not be able to reproduce exactly. However without your model, we cannot even diagnose what is wrong. We can have a look at your model and spot hot points in it.

If you don't want to share your model in public, please send it to me in email (I will share it with dev team members only).

Thanks

vlkong commented 3 years ago

@Milad321 Additionnaly, you can try to build your model without attaching it to CPLEX, then export to .lp and send me the model (by email) ? Replace your model creation code with:

model = Model(agent='zero')

then once your model is built:

model.export_as_lp("model.lp")

Milad321 commented 3 years ago

@Milad321 Hi Milad. Can you provide use the model ? I understand that because input data is from your simulation environment, we might not be able to reproduce exactly. However without your model, we cannot even diagnose what is wrong. We can have a look at your model and spot hot points in it.

If you don't want to share your model in public, please send it to me in email (I will share it with dev team members only).

Thanks

Thank you for your immediate response. Yes, I will send my model to you in private.

Sincerely,

vlkong commented 3 years ago

Confirming here that I received the python model, without data (too large). Team is having a look at the model.

Milad321 commented 3 years ago

Thank you so much.

vlkong commented 3 years ago

To sum up:

The problem @Milad321 tries to create is very large, using 42 GB of virtual memory. We see that the process fails at 50% of the free memory, because Windows process explorer only displays the physical memory. So there is no bug in CPLEX.
The original model generated thousands (millions) of temp expressions - Garbage collector is taking all the time. Replacing some loops with mdl.sum() bring some phases of the model construction time from about 1h to 40 seconds. And more memory is available to CPLEX.
Even with some code optimization, the model as it is expressed is too large for docplex + CPLEX ( docplex add a lot of management around CPLEX)
@Milad321 will try to express is model using the CPLEX matrix api, which should save a lot of memory needed to map python objects to matrix.

Thanks