Problem solving in multithreading

[nishit727]

Hi, I am trying to solve a stochastic problem. The problem for 24 steps and in each step I solve it for 450 different instances. Now it is possible to solve the problem serially, but that takes a lot of time, so i wanted to solve those 450 instances parallely. I approached this in 2 ways: 1) using solver_manager and multiple pyro mip servers. Here I just have to queue the insances and the solver manager would assign the problems to pyro mip servers. But the pyro mip servers occupy a lot of memory and retain it even after the final result is computed. 2) using opt solver. a) I can solve using the opt solver directly, but it is slow. b) using multi threading. I tried to create a threadPoolExecutor with 5 threads and then solve those 450 instances in different threads. But just shifting to multithreading causes the pyomo to throw errors.

Code for this approach - ` def optimize():

optsolver = SolverFactory(self.solver_name)
value = self.initialize_all_values()
for timestep in range(24):
    instance_list = self.create_instance_list(value)
    futures = {}
    with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
        for instance_object in instance_list:
            futures = {executor.submit(self.thread_solver, instance_object["instance"], optsolver, timestep)}
        for future in concurrent.futures.as_completed(futures):
            try:
                result = future.result()
                value.update(result)
            except Exception as e:
                self.logger.error(e)

def thread_solver(instance, optsolver, timestep):

result = optsolver.solve(instance)
if (result.solver.status == SolverStatus.ok) and (
        result.solver.termination_condition == TerminationCondition.optimal):
    instance.solutions.load_from(result)\
    my_dict = {}
    for v in instance.component_objects(Var, active=True):
        varobject = getattr(instance, str(v))
        var_list = []
        try:
            for index in varobject:
                var_list.append(varobject[index].value)
            my_dict[str(v)] = var_list
        except Exception as e:
            self.logger.error("error reading result " + str(e))
    value = {key: my_dict[k][0]}
    return value
elif result.solver.termination_condition == TerminationCondition.infeasible:
    self.logger.info("Termination condition is infeasible")
    return {}
else:
    self.logger.info("Nothing fits")
    return {}

` I get the following errors/outputs: 1) Nothing fits 2) Solver failed to locate input problem file: /usr/src/app/temp/pyomo/tmpn23yhmz_.pyomo.lp I have set the pyomo temp file location to "/usr/src/app/temp/pyomo"

The same code runs perfectly fine without multithreading. Any reason for the pyomo to not solve the problem?

[blnicho]

Issues on the Pyomo GitHub page are meant for bug-reports and feature requests. If you have modeling or implementation questions please ask them on the Pyomo forum or on StackOverflow.

[yifanguan]

Hi all, I believe this is a Pyomo bug or feature needed to be added to Pyomo because in multi-threading scenarios, if we want to solve multiple models in parallel. Parallel calls of results = solver.solve(model) generate an error: Solver failed to locate input problem file: /tmp/tmpvk4cpr9c.pyomo.lp

[rhshadrach]

I am experiencing the same issue, I can solve multiple models using multiprocessing successfully. However, I'd like to solve them using multithreading to avoid the serialization overhead. However, I get a similar error as @yifanguan. My pyomo version is 5.7.3.