to discuss: catch ApplicationError returned from ipopt call?

[rjhanes]

Running commit 3f8d59f (array-uncertainty code repo branch) on commit 150f92d (national-array-uncertainty data repo branch), using the scenario.yaml file in the data branch, I ran into a "Solver (ipopt) returned non-zero return code" error that stopped execution. Complete error output is below. I don't think this is a problem with CELAVI or the data but with IPOPT, and although this is a relatively rare occurrence it could nonetheless cause problems with longer HPC uncertainty runs.

I suggest implementing try/except statements around IPOPT calls, such that if the solver chokes, the optimization is either attempted again or all zeros are returned with a printed warning message. That way CELAVI can keep running with minimal or no impact on the final results.

On the other hand, if IPOPT fails so rarely that it's not likely to cause complications with HPC uncertainty runs, then I don't think a fix would add much value.

@TJTapajyoti , do you think implementing this fix would be beneficial and worth the time it would take?

@akey7 and @jwalzberg , have you seen this error or a similar error in your own testing, and what are your thoughts on implementing this fix?

Updating costs for 2023 at         1965 s
Costs updated for  2023 at         1965 s
2022-05-25 13:50:56.039116 pylca_interface_process(): Found flow quantities greater than 0, performing LCIA
ERROR: Solver (ipopt) returned non-zero return code (3221225595)
Traceback (most recent call last):
  File "C:\Users\rhanes\GitHub\celavi\celavi\des.py", line 485, in pylca_interface_process
    self.lca.pylca_run_main(df_for_pylca_interface)
  File "C:\Users\rhanes\GitHub\celavi\celavi\pylca_celavi\des_interface.py", line 284, in pylca_run_main
    emission = model_celavi_lci_insitu(
  File "C:\Users\rhanes\GitHub\celavi\celavi\pylca_celavi\insitu_emission.py", line 275, in model_celavi_lci_insitu
    res = runner(tech_matrix, F, yr, fac_id, stage, material, 100000, process, df_with_all_other_flows)
  File "C:\Users\rhanes\GitHub\celavi\celavi\pylca_celavi\insitu_emission.py", line 201, in runner
    res= solver_optimization(tech_matrix, F,process,df_with_all_other_flows)
  File "C:\Users\rhanes\GitHub\celavi\celavi\pylca_celavi\insitu_emission.py", line 127, in solver_optimization
    results = opt.solve(model)
  File "C:\Users\rhanes\AppData\Local\Continuum\anaconda3\envs\celavi\lib\site-packages\pyomo\opt\base\solvers.py", line 601, in solve
    raise ApplicationError(
pyutilib.common._exceptions.ApplicationError: Solver (ipopt) did not exit normally

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\rhanes\GitHub\celavi\celavi\__main__.py", line 25, in <module>
    Scenario(parser=PARSER)
  File "C:\Users\rhanes\GitHub\celavi\celavi\scenario.py", line 113, in __init__
    self.execute()
  File "C:\Users\rhanes\GitHub\celavi\celavi\scenario.py", line 468, in execute
    self.context.run()
  File "C:\Users\rhanes\GitHub\celavi\celavi\des.py", line 548, in run
    self.env.run(until=int(self.max_timesteps))
  File "C:\Users\rhanes\AppData\Local\Continuum\anaconda3\envs\celavi\lib\site-packages\simpy\core.py", line 254, in run
    self.step()
  File "C:\Users\rhanes\AppData\Local\Continuum\anaconda3\envs\celavi\lib\site-packages\simpy\core.py", line 206, in step
    raise exc
pyutilib.common._exceptions.ApplicationError: Solver (ipopt) did not exit normally

[rjhanes]

In trying to reproduce issues #162 and #163 (code commit 93fcf87, data commit e2de961, 1_cglr.yaml from OneDrive), I ran into this error again at the same point in the simulation. While IPOPT is choking on something, the repeat error would indicate that there's a problem with what's being fed into the solver from the insitu emissions method (see full errror code above).

Debugging process:

I inserted a pd.set_trace() at line 127 in insitu_emission.py and confirmed that IPOPT is returning the non-zero code the first time this method is being used during a simulation. I explored the contents of model and the parameters being fed into the method but didn't see anything obviously wrong.

Second attempt: After line 126 in insitu_emission.py, I replaced

results = opt.solve(model)
solution = pyomo_postprocess(None, model, results)

with

    try:
        results = opt.solve(model)
        solution = pyomo_postprocess(None, model, results)
    except:
        print(f"Solver (ipopt) did not exit normally; skipping LCIA for processes {process}")
        solution = pyomo_postprocess()

Solution: Swapping out ipopt for glpk to see if that fixes the issues

[rjhanes]

Resolved with commit 796ed0e