Context: in cmu-delphi/flu-hosp-forecast, we use a Python CLI script to call our R forecasting scripts. We tested out using rpy2 in those scripts, resulting in apparent nontermination on some systems for larger models. Debugging revealed a few interesting behaviors:
gurobi will automatically parallelize on larger problems, even without a Threads setting
gurobi parallelization may cause nontermination on some systems when combined with some libraries (such as rpy2)
on some problems, Threads=2 gives a 16x speedup compared to Threads=1! And the default Threads, which for these problems used several more than 2 cores but many less than 16, still gave basically the same 16x speedup. Looking at this in reverse, using Threads=1 to try to address rpy2 incompatibility isn't ideal due to its slowness.
Ideally there'd be some way to programmatically detect and warn/adjust when the nontermination behavior's going to arise (and what the ideal number of threads would be), but we haven't completely pinned it down yet. Documentation edits might help. (Or maybe just having this issue filed will help others encountering incompatibilities to debug.)
Context: in cmu-delphi/flu-hosp-forecast, we use a Python CLI script to call our R forecasting scripts. We tested out using rpy2 in those scripts, resulting in apparent nontermination on some systems for larger models. Debugging revealed a few interesting behaviors:
Ideally there'd be some way to programmatically detect and warn/adjust when the nontermination behavior's going to arise (and what the ideal number of threads would be), but we haven't completely pinned it down yet. Documentation edits might help. (Or maybe just having this issue filed will help others encountering incompatibilities to debug.)