Unable to use SCIP as solver with Pyomo

Hi Nathan @spiralulam

I cloned the master branch and did pip install . then installed glpk and scip via conda as written in the docs. Then I realized I still did not have pyomo or gurobipy, so I did pip install -r requirements.txt and everything seemed good to go. But when I run multi_obj_with_constraints.ipynb (the first one I tried) I got errors with both the Gurobi and pyomo versions at the lines res_gur = opt_gur.solve(enting, model_core=model_gur) and / or res_pyo = opt_pyo.solve(enting, model_core=model_pyo)

The error (for the pyomo version) is:

AttributeError                            Traceback (most recent call last)
Untitled-1 in line 1
----> [88](untitled:Untitled-1?line=87) res_pyo = opt_pyo.solve(enting, model_core=model_pyo)

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\optimizers\pyomo_opt.py:96](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/optimizers/pyomo_opt.py:96), in PyomoOptimizer.solve(self, tree_model, model_core, weights)
     [93](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/optimizers/pyomo_opt.py?line=92)         opt.options[k] = v
     [95](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/optimizers/pyomo_opt.py?line=94) # build pyomo model using information from tree model
---> [96](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/optimizers/pyomo_opt.py?line=95) tree_model.add_to_pyomo_model(opt_model)
     [98](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/optimizers/pyomo_opt.py?line=97) # Solve optimization model
     [99](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/optimizers/pyomo_opt.py?line=98) opt.solve(opt_model, tee=True)

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\enting.py:209](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py:209), in Enting.add_to_pyomo_model(self, core_model, weights)
    [204](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=203)     core_model.constraint_link_mu_auxmu = pyo.Constraint(
    [205](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=204)         expr=core_model._aux_mu[obj_name] == core_model._mu
    [206](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=205)     )
    [207](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=206) else:
    [208](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=207)     # multi-objective case
--> [209](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=208)     self.mean_model.add_to_pyomo_model(
    [210](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=209)         core_model, add_mu_var=True, normalize_mean=True
    [211](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=210)     )
    [212](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=211)     if weights is not None:
    [213](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/enting.py?line=212)         moo_weights = weights

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\tree_ensemble.py:402](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py:402), in TreeEnsemble.add_to_pyomo_model(self, model, add_mu_var, normalize_mean)
    [397](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=396) def add_to_pyomo_model(
    [398](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=397)     self, model, add_mu_var: bool = True, normalize_mean: bool = False
    [399](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=398) ):
    [400](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=399)     import pyomo.environ as pyo
--> [402](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=401)     self._update_meta_tree_dict()
    [404](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=403)     # attach tree info
    [405](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=404)     model._num_trees = lambda obj_name: self._meta_tree_dict[obj_name].num_trees

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\tree_ensemble.py:132](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py:132), in TreeEnsemble._update_meta_tree_dict(self)
    [126](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=125)     raise IOError(
    [127](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=126)         "Parameter 'train_lib' for tree ensembles needs to be "
    [128](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=127)         "in '('lgbm')'."
    [129](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=128)     )
    [131](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=130) # order tree_model_dict
--> [132](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=131) ordered_tree_model_dict = read_lgbm_tree_model_dict(
    [133](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=132)     lib_out, cat_idx=self._problem_config.cat_idx
    [134](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=133) )
    [136](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=135) # populate meta_tree_model
    [137](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/tree_ensemble.py?line=136) self._meta_tree_dict[obj.name] = MetaTreeModel(ordered_tree_model_dict)

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:11](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py:11), in read_lgbm_tree_model_dict(tree_model_dict, cat_idx)
      [8](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=7) node_list = []
     [10](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=9) # populate node_list and add to ordered_tree_list if non-empty
---> [11](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=10) add_next_nodes(node_list=node_list, node=root_node, cat_idx=cat_idx)
     [13](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=12) if node_list:
     [14](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=13)     ordered_tree_list.append(node_list)

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:56](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py:56), in add_next_nodes(node_list, node, cat_idx)
     [53](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=52) except KeyError:
     [54](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=53)     pass
---> [56](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=55) add_next_nodes(node_list, node, cat_idx=cat_idx)

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:56](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py:56), in add_next_nodes(node_list, node, cat_idx)
     [53](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=52) except KeyError:
     [54](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=53)     pass
---> [56](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=55) add_next_nodes(node_list, node, cat_idx=cat_idx)

    [... skipping similar frames: add_next_nodes at line 56 (3 times)]

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:56](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py:56), in add_next_nodes(node_list, node, cat_idx)
     [53](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=52) except KeyError:
     [54](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=53)     pass
---> [56](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=55) add_next_nodes(node_list, node, cat_idx=cat_idx)

File [c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:32](file:///C:/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py:32), in add_next_nodes(node_list, node, cat_idx)
     [29](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=28)     new_node["split_code_pred"] = temp_node_val
     [30](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=29) else:
     [31](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=30)     # read categorical variables
---> [32](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=31)     cat_set = node[-1]["threshold"].split("||")
     [33](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=32)     temp_node_val = [int(cat) for cat in cat_set]
     [34](file:///c%3A/Users/myuser/dev/entmoot_test/entmoot/entmoot/models/mean_models/lgbm_utils.py?line=33)     new_node["split_code_pred"] = temp_node_val

AttributeError: 'float' object has no attribute 'split'`

Any idea what is missing here?

Thanks, Robert. I will look into it. Best, Nathan

Am Do., 23. Nov. 2023 um 10:03 Uhr schrieb Robert Lee < @.***>:

Hi Nathan @spiralulam https://github.com/spiralulam

I cloned the master branch and did pip install . then installed glpk and scip via conda as written in the docs. Then I realized I still did not have pyomo or gurobipy, so I did pip install -r requirements.txt and everything seemed good to go. But when I run multi_obj_with_constraints.ipynb (the first one I tried) I got errors with both the Gurobi and pyomo versions at the lines res_gur = opt_gur.solve(enting, model_core=model_gur) and / or res_pyo = opt_pyo.solve(enting, model_core=model_pyo)

The error (for the pyomo version) is:

AttributeError Traceback (most recent call last) Untitled-1 in line 1 ----> 88 res_pyo = opt_pyo.solve(enting, model_core=model_pyo)

File c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\optimizers\pyomo_opt.py:96, in PyomoOptimizer.solve(self, tree_model, model_core, weights) 93 opt.options[k] = v 95 # build pyomo model using information from tree model ---> 96 tree_model.add_to_pyomo_model(opt_model) 98 # Solve optimization model 99 opt.solve(opt_model, tee=True)

File c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\enting.py:209, in Enting.add_to_pyomo_model(self, core_model, weights) 204 core_model.constraint_link_mu_auxmu = pyo.Constraint( 205 expr=core_model._aux_mu[obj_name] == core_model._mu 206 ) 207 else: 208 # multi-objective case --> 209 self.mean_model.add_to_pyomo_model( 210 core_model, add_mu_var=True, normalize_mean=True 211 ) 212 if weights is not None: 213 moo_weights = weights

File c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\tree_ensemble.py:402, in TreeEnsemble.add_to_pyomo_model(self, model, add_mu_var, normalize_mean) 397 def add_to_pyomo_model( 398 self, model, add_mu_var: bool = True, normalize_mean: bool = False 399 ): 400 import pyomo.environ as pyo --> 402 self._update_meta_tree_dict() 404 # attach tree info 405 model._num_trees = lambda obj_name: self._meta_tree_dict[obj_name].num_trees

File c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\tree_ensemble.py:132, in TreeEnsemble._update_meta_tree_dict(self) 126 raise IOError( 127 "Parameter 'train_lib' for tree ensembles needs to be " 128 "in '('lgbm')'." 129 ) 131 # order tree_model_dict --> 132 ordered_tree_model_dict = read_lgbm_tree_model_dict( 133 lib_out, cat_idx=self._problem_config.cat_idx 134 ) 136 # populate meta_tree_model 137 self._meta_tree_dict[obj.name] = MetaTreeModel(ordered_tree_model_dict)

File c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:11, in read_lgbm_tree_model_dict(tree_model_dict, cat_idx) 8 node_list = [] 10 # populate node_list and add to ordered_tree_list if non-empty ---> 11 add_next_nodes(node_list=node_list, node=root_node, cat_idx=cat_idx) 13 if node_list: 14 ordered_tree_list.append(node_list)

File c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:56, in add_next_nodes(node_list, node, cat_idx) 53 except KeyError: 54 pass ---> 56 add_next_nodes(node_list, node, cat_idx=cat_idx)

File c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:56, in add_next_nodes(node_list, node, cat_idx) 53 except KeyError: 54 pass ---> 56 add_next_nodes(node_list, node, cat_idx=cat_idx)
[... skipping similar frames: add_next_nodes at line 56 (3 times)]
File c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:56, in add_next_nodes(node_list, node, cat_idx) 53 except KeyError: 54 pass ---> 56 add_next_nodes(node_list, node, cat_idx=cat_idx)

File c:\Users\myuser\dev\entmoot_test\entmoot\entmoot\models\mean_models\lgbm_utils.py:32, in add_next_nodes(node_list, node, cat_idx) 29 new_node["split_code_pred"] = temp_node_val 30 else: 31 # read categorical variables ---> 32 cat_set = node[-1]["threshold"].split("||") 33 temp_node_val = [int(cat) for cat in cat_set] 34 new_node["split_code_pred"] = temp_node_val

AttributeError: 'float' object has no attribute 'split'`

Any idea what is missing here?

— Reply to this email directly, view it on GitHub https://github.com/cog-imperial/entmoot/issues/28, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK3L6OBMEOZZF6PPFXUMAGDYF4GMXAVCNFSM6AAAAAA7XNOREOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGAYDONZVGY3DCNA . You are receiving this because you were mentioned.Message ID: @.***>

I fixed now that you had to install gurobipy/pyomo. This has to be specified in setup.py, I thought there would be a reference to requirements.txt, but there is a very lengthy and well rated post (that I haven't read) on stackoverflow where someone explains that this duplication makes sense because requirements.txt and setup.py serve different purposes.

It should work now, please test again, @R-M-Lee . The problem was the lightgbm version which is now fixed to 4.0.0. Will open an issue for that one.

Thanks for looking into this. Now it runs with glpk and gurobi, but there is something wrong with the scip installation (I did it via conda as described in the docs)

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
c:\Users\user\dev\entmoot_test\entmoot\docs\notebooks\multi_obj_with_constraints.ipynb Cell 15 line 1
     14 params_pyomo = {"solver_name": "SCIP"}
     15 opt_pyo = PyomoOptimizer(problem_config, params=params_pyomo)
---> 17 res_pyo = opt_pyo.solve(enting, model_core=model_pyo)
     18 x_opt, y_opt, z_opt = res_pyo.opt_point[3:]
     20 assert round(x_opt, 5) == round(y_opt, 5) and round(y_opt, 5) == round(z_opt, 5)

File c:\Users\user\Miniconda3\envs\entmoottest2\lib\site-packages\entmoot\optimizers\pyomo_opt.py:99, in PyomoOptimizer.solve(self, tree_model, model_core, weights)
     96 tree_model.add_to_pyomo_model(opt_model)
     98 # Solve optimization model
---> 99 opt.solve(opt_model, tee=True)
    101 # update current solution
    102 self._curr_sol, self._active_leaves = self._get_sol(opt_model)

File c:\Users\user\Miniconda3\envs\entmoottest2\lib\site-packages\pyomo\opt\base\solvers.py:635, in OptSolver.solve(self, *args, **kwds)
    629 if self._report_timing:
    630     print(
    631         "      %6.2f seconds required for solver"
    632         % (solve_completion_time - presolve_completion_time)
    633     )
--> 635 result = self._postsolve()
    636 result._smap_id = self._smap_id
    637 result._smap = None

File c:\Users\user\Miniconda3\envs\entmoottest2\lib\site-packages\pyomo\solvers\plugins\solvers\ASL.py:230, in ASL._postsolve(self)
    228 #
    229 self._instance = None
--> 230 return SystemCallSolver._postsolve(self)

File c:\Users\user\Miniconda3\envs\entmoottest2\lib\site-packages\pyomo\opt\solver\shellcmd.py:291, in SystemCallSolver._postsolve(self)
    288 results = None
    290 if self._results_format is not None:
--> 291     results = self.process_output(self._rc)
    292     #
    293     # If keepfiles is true, then we pop the
    294     # TempfileManager context while telling it to
    295     # _not_ remove the files.
    296     #
    297     if not self._keepfiles:
    298         # in some cases, the solution filename is
    299         # not generated via the temp-file mechanism,
    300         # instead being automatically derived from
    301         # the input lp/nl filename. so, we may have
    302         # to clean it up manually.

File c:\Users\user\Miniconda3\envs\entmoottest2\lib\site-packages\pyomo\opt\solver\shellcmd.py:396, in SystemCallSolver.process_output(self, rc)
    389     results = self._results_reader(
    390         self._results_file,
    391         res=results,
    392         soln=results.solution(0),
    393         suffixes=self._suffixes,
    394     )
    395 else:
--> 396     results = self._results_reader(
    397         self._results_file, res=results, suffixes=self._suffixes
    398     )
    399 results_reader_completion_time = time.time()
    400 if self._report_timing is True:

File c:\Users\user\Miniconda3\envs\entmoottest2\lib\site-packages\pyomo\opt\plugins\sol.py:40, in ResultsReader_sol.__call__(self, filename, res, soln, suffixes)
     36 """
     37 Parse a *.sol file
     38 """
     39 try:
---> 40     with open(filename, "r") as f:
     41         return self._load(f, res, soln, suffixes)
     42 except ValueError as e:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\user\\AppData\\Local\\Temp\\tmpi0kx1olr.pyomo.sol'

There seems to be no way to use SCIP after a conda installation without changing some code snippets. I traced it down to this part of Pyomo's code: Up to now, we use SolverFactory("gurobi"), SolverFactory("glpk") and so on... BUT scip requires the use of a different interface, namely SolverFactory(solver='scip', solver_io='nl'), see here (comment of Matthias Fripp https://stackoverflow.com/questions/56885640/problems-in-interfacing-scip-with-pyomo/56886542#56886542) and this is unfortunately handled in a very different way (see screenshot above). So unfortunately no SCIP for now...

cog-imperial / entmoot

Unable to use SCIP as solver with Pyomo #28