Closed jurasofish closed 4 years ago
Hi,
Interesting, I always used the default start methods for multiprocessing (fork in linux). Just reading the documentation now, it seems that fork gives some problems in MacOS, they really recommend spawn:
"Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. "
https://docs.python.org/3/library/multiprocessing.html
On Mon, 1 Jun 2020 at 21:48, Michael Jurasovic notifications@github.com wrote:
Hi, I've found a case where mip hangs while optimising the model, but only in multiprocessing. Changing the multiprocessing mode to spawn fixes the issue. I'm running this on MacOS which means the default multiprocessing method is fork.
I'll see if I can produce a minimal working example. Haven't been able to so far, and saving then restoring the model from an lp or mps file doesn't reproduce the issue.
Works:
import miprun_model()
hangs:
import mipfrom multiprocessing import Pool def main():
futures = [] with Pool() as p: futures.append(p.apply_async(run_model, ())) for future in futures: print(future.get())
if name == 'main': main()
Works:
import mipfrom multiprocessing import Poolimport multiprocessing as mp def main():
futures = [] with Pool() as p: futures.append(p.apply_async(run_model, ())) for future in futures: print(future.get())
if name == 'main': mp.set_start_method('spawn') main()
Model verbose output when it works:
Starting solution of the Linear programming relaxation problem using Dual Simplex
Coin0506I Presolve 284 (-203) rows, 209 (-36) columns and 908 (-446) elements Clp0006I 0 Obj 4.1243524 Primal inf 1203.3021 (49) Dual inf 5.0110246 (158) Clp0000I Optimal - objective value 1618.625 Coin0511I After Postsolve, objective 1618.625, infeasibilities - dual 0 (0), primal 0 (0) Clp0032I Optimal objective 1618.625024 - 156 iterations time 0.002, Presolve 0.00
Starting MIP optimization maxSavedSolutions was changed from 0 to 10 Continuous objective value is 1618.63 - 0.00 seconds Cgl0003I 0 fixed, 0 tightened bounds, 1 strengthened rows, 0 substitutions Cgl0004I processed model has 286 rows, 210 columns (1 integer (1 of which binary)) and 916 elements Coin3009W Conflict graph built in 0.000 seconds, density: 0.001% Cbc0038I Initial state - 1 integers unsatisfied sum - 0.222301 Cbc0038I Pass 1: suminf. 0.00000 (0) obj. -1553.08 iterations 4 Cbc0038I Solution found of -1553.08 Cbc0038I Relaxing continuous gives -1557.95 Cbc0038I Before mini branch and bound, 0 integers at bound fixed and 91 continuous Cbc0038I Full problem 286 rows 210 columns, reduced to 28 rows 28 columns Cbc0038I Mini branch and bound improved solution from -1557.95 to -1618.62 (0.02 seconds) Cbc0038I Freeing continuous variables gives a solution of -1618.62 Cbc0038I Round again with cutoff of -1618.63 Cbc0038I Pass 2: suminf. 0.22230 (1) obj. -1618.63 iterations 6 Cbc0038I Pass 3: suminf. 0.00000 (1) obj. -1618.63 iterations 4 Cbc0038I Pass 4: suminf. 0.00000 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 5: suminf. 0.22230 (1) obj. -1618.63 iterations 1 Cbc0038I Pass 6: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 7: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 8: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 9: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 10: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 11: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 12: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 13: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 14: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 15: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 16: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 17: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 18: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 19: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 20: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 21: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 22: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 23: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 24: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 25: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 26: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 27: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 28: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 29: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 30: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 31: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I No solution found this major pass Cbc0038I Before mini branch and bound, 0 integers at bound fixed and 93 continuous Cbc0038I Full problem 286 rows 210 columns, reduced to 8 rows 10 columns Cbc0038I Mini branch and bound did not improve solution (0.03 seconds) Cbc0038I After 0.03 seconds - Feasibility pump exiting with objective of -1618.62 - took 0.01 seconds Cbc0012I Integer solution of -1618.625 found by feasibility pump after 0 iterations and 0 nodes (0.03 seconds) Cbc0011I Exiting as integer gap of 2.84888e-05 less than 1e-10 or 0.01% Cbc0001I Search completed - best objective -1618.624995303767, took 0 iterations and 0 nodes (0.03 seconds) Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost Cuts at root node changed objective from -1618.63 to -1618.63 Probing was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) Gomory was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) Knapsack was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) Clique was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) OddWheel was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) MixedIntegerRounding2 was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) FlowCover was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) TwoMirCuts was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) ZeroHalf was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
Result - Optimal solution found (within gap tolerance)
Objective value: 1618.62499530 Upper bound: 1618.625 Gap: -0.00 Enumerated nodes: 0 Total iterations: 0 Time (CPU seconds): 0.03 Time (Wallclock seconds): 0.05
Total time (CPU seconds): 0.03 (Wallclock seconds): 0.05
Model verbose output when it hangs:
Starting solution of the Linear programming relaxation problem using Dual Simplex
Coin0506I Presolve 284 (-203) rows, 209 (-36) columns and 908 (-446) elements Clp0006I 0 Obj 4.1243524 Primal inf 1203.3021 (49) Dual inf 5.0110246 (158) Clp0000I Optimal - objective value 1618.625 Coin0511I After Postsolve, objective 1618.625, infeasibilities - dual 0 (0), primal 0 (0) Clp0032I Optimal objective 1618.625024 - 156 iterations time 0.002, Presolve 0.00
Starting MIP optimization maxSavedSolutions was changed from 0 to 10 Continuous objective value is 1618.63 - 0.00 seconds Cgl0003I 0 fixed, 0 tightened bounds, 1 strengthened rows, 0 substitutions Cgl0004I processed model has 286 rows, 210 columns (1 integer (1 of which binary)) and 916 elements Coin3009W Conflict graph built in 0.000 seconds, density: 0.001% Cbc0038I Initial state - 1 integers unsatisfied sum - 0.222301 Cbc0038I Pass 1: suminf. 0.00000 (0) obj. -1553.08 iterations 4 Cbc0038I Solution found of -1553.08 Cbc0038I Relaxing continuous gives -1557.95 Cbc0038I Before mini branch and bound, 0 integers at bound fixed and 91 continuous
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/coin-or/python-mip/issues/116, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4VZOSPUA4X5HXCORKUOPDRURD4JANCNFSM4NQIOFUA .
Haroldo Gambini Santos Computing Department Universidade Federal de Ouro Preto - UFOP email: haroldo@ufop.edu.br home/research page: www.decom.ufop.br/haroldo
It has long been an axiom of mine that the little things are infinitely the most important. -- Sir Arthur Conan Doyle, "A Case of Identity"
forkserver seems to be an alternative too
On Mon, 1 Jun 2020 at 22:06, HAROLDO GAMBINI SANTOS haroldo@ufop.edu.br wrote:
Hi,
Interesting, I always used the default start methods for multiprocessing (fork in linux). Just reading the documentation now, it seems that fork gives some problems in MacOS, they really recommend spawn:
"Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. "
https://docs.python.org/3/library/multiprocessing.html
On Mon, 1 Jun 2020 at 21:48, Michael Jurasovic notifications@github.com wrote:
Hi, I've found a case where mip hangs while optimising the model, but only in multiprocessing. Changing the multiprocessing mode to spawn fixes the issue. I'm running this on MacOS which means the default multiprocessing method is fork.
I'll see if I can produce a minimal working example. Haven't been able to so far, and saving then restoring the model from an lp or mps file doesn't reproduce the issue.
Works:
import miprun_model()
hangs:
import mipfrom multiprocessing import Pool def main():
futures = [] with Pool() as p: futures.append(p.apply_async(run_model, ())) for future in futures: print(future.get())
if name == 'main': main()
Works:
import mipfrom multiprocessing import Poolimport multiprocessing as mp def main():
futures = [] with Pool() as p: futures.append(p.apply_async(run_model, ())) for future in futures: print(future.get())
if name == 'main': mp.set_start_method('spawn') main()
Model verbose output when it works:
Starting solution of the Linear programming relaxation problem using Dual Simplex
Coin0506I Presolve 284 (-203) rows, 209 (-36) columns and 908 (-446) elements Clp0006I 0 Obj 4.1243524 Primal inf 1203.3021 (49) Dual inf 5.0110246 (158) Clp0000I Optimal - objective value 1618.625 Coin0511I After Postsolve, objective 1618.625, infeasibilities - dual 0 (0), primal 0 (0) Clp0032I Optimal objective 1618.625024 - 156 iterations time 0.002, Presolve 0.00
Starting MIP optimization maxSavedSolutions was changed from 0 to 10 Continuous objective value is 1618.63 - 0.00 seconds Cgl0003I 0 fixed, 0 tightened bounds, 1 strengthened rows, 0 substitutions Cgl0004I processed model has 286 rows, 210 columns (1 integer (1 of which binary)) and 916 elements Coin3009W Conflict graph built in 0.000 seconds, density: 0.001% Cbc0038I Initial state - 1 integers unsatisfied sum - 0.222301 Cbc0038I Pass 1: suminf. 0.00000 (0) obj. -1553.08 iterations 4 Cbc0038I Solution found of -1553.08 Cbc0038I Relaxing continuous gives -1557.95 Cbc0038I Before mini branch and bound, 0 integers at bound fixed and 91 continuous Cbc0038I Full problem 286 rows 210 columns, reduced to 28 rows 28 columns Cbc0038I Mini branch and bound improved solution from -1557.95 to -1618.62 (0.02 seconds) Cbc0038I Freeing continuous variables gives a solution of -1618.62 Cbc0038I Round again with cutoff of -1618.63 Cbc0038I Pass 2: suminf. 0.22230 (1) obj. -1618.63 iterations 6 Cbc0038I Pass 3: suminf. 0.00000 (1) obj. -1618.63 iterations 4 Cbc0038I Pass 4: suminf. 0.00000 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 5: suminf. 0.22230 (1) obj. -1618.63 iterations 1 Cbc0038I Pass 6: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 7: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 8: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 9: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 10: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 11: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 12: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 13: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 14: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 15: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 16: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 17: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 18: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 19: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 20: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 21: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 22: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 23: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 24: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 25: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 26: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 27: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 28: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 29: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 30: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I Pass 31: suminf. 0.22230 (1) obj. -1618.63 iterations 0 Cbc0038I No solution found this major pass Cbc0038I Before mini branch and bound, 0 integers at bound fixed and 93 continuous Cbc0038I Full problem 286 rows 210 columns, reduced to 8 rows 10 columns Cbc0038I Mini branch and bound did not improve solution (0.03 seconds) Cbc0038I After 0.03 seconds - Feasibility pump exiting with objective of -1618.62 - took 0.01 seconds Cbc0012I Integer solution of -1618.625 found by feasibility pump after 0 iterations and 0 nodes (0.03 seconds) Cbc0011I Exiting as integer gap of 2.84888e-05 less than 1e-10 or 0.01% Cbc0001I Search completed - best objective -1618.624995303767, took 0 iterations and 0 nodes (0.03 seconds) Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost Cuts at root node changed objective from -1618.63 to -1618.63 Probing was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) Gomory was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) Knapsack was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) Clique was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) OddWheel was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) MixedIntegerRounding2 was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) FlowCover was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) TwoMirCuts was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) ZeroHalf was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
Result - Optimal solution found (within gap tolerance)
Objective value: 1618.62499530 Upper bound: 1618.625 Gap: -0.00 Enumerated nodes: 0 Total iterations: 0 Time (CPU seconds): 0.03 Time (Wallclock seconds): 0.05
Total time (CPU seconds): 0.03 (Wallclock seconds): 0.05
Model verbose output when it hangs:
Starting solution of the Linear programming relaxation problem using Dual Simplex
Coin0506I Presolve 284 (-203) rows, 209 (-36) columns and 908 (-446) elements Clp0006I 0 Obj 4.1243524 Primal inf 1203.3021 (49) Dual inf 5.0110246 (158) Clp0000I Optimal - objective value 1618.625 Coin0511I After Postsolve, objective 1618.625, infeasibilities - dual 0 (0), primal 0 (0) Clp0032I Optimal objective 1618.625024 - 156 iterations time 0.002, Presolve 0.00
Starting MIP optimization maxSavedSolutions was changed from 0 to 10 Continuous objective value is 1618.63 - 0.00 seconds Cgl0003I 0 fixed, 0 tightened bounds, 1 strengthened rows, 0 substitutions Cgl0004I processed model has 286 rows, 210 columns (1 integer (1 of which binary)) and 916 elements Coin3009W Conflict graph built in 0.000 seconds, density: 0.001% Cbc0038I Initial state - 1 integers unsatisfied sum - 0.222301 Cbc0038I Pass 1: suminf. 0.00000 (0) obj. -1553.08 iterations 4 Cbc0038I Solution found of -1553.08 Cbc0038I Relaxing continuous gives -1557.95 Cbc0038I Before mini branch and bound, 0 integers at bound fixed and 91 continuous
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/coin-or/python-mip/issues/116, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4VZOSPUA4X5HXCORKUOPDRURD4JANCNFSM4NQIOFUA .
--
Haroldo Gambini Santos Computing Department Universidade Federal de Ouro Preto - UFOP email: haroldo@ufop.edu.br home/research page: www.decom.ufop.br/haroldo
It has long been an axiom of mine that the little things are infinitely the most important. -- Sir Arthur Conan Doyle, "A Case of Identity"
Haroldo Gambini Santos Computing Department Universidade Federal de Ouro Preto - UFOP email: haroldo@ufop.edu.br home/research page: www.decom.ufop.br/haroldo
It has long been an axiom of mine that the little things are infinitely the most important. -- Sir Arthur Conan Doyle, "A Case of Identity"
Hi, Interesting, I always used the default start methods for multiprocessing (fork in linux). Just reading the documentation now, it seems that fork gives some problems in MacOS, they really recommend spawn: "Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. " https://docs.python.org/3/library/multiprocessing.html
That's good to know, I hadn't come across that, thanks.
I should mention as well this is a pretty obscure bug - I've run hundreds of very very similar models with similar structure and similar values with no issue.
I think we agree just don't use fork on macos so I'll close
Hi, I've found a case where mip hangs while optimising the model, but only in multiprocessing. Changing the multiprocessing mode to spawn fixes the issue. I'm running this on MacOS which means the default multiprocessing method is fork.
I'll see if I can produce a minimal working example. Haven't been able to so far, and saving then restoring the model from an lp or mps file doesn't reproduce the issue.
Works:
hangs:
Works:
Model verbose output when it works:
Model verbose output when it hangs: