Feature: Gurobi backend interface

[brynpickering]

Fixes #349 Fixes #597 (by pinning pyomo)

Summary of changes in this pull request:

• Duplicated pyomo backend interface as a pure gurobipy interface
• Added basic tests of the interface methods

How to use:

1. Install gurobipy into your calliope env: mamba install gurobi::gurobi
2. Build optimisation problem with gurobi backend:

   
   import calliope

model = calliope.Model(...) model.build(backend="gurobi") model.solve()

NOTE: without a license, one can only work with _very_ small models. Our example models are already too big (maybe if run over ~2 timesteps it would work?).

Reviewer checklist:

- [x] Test(s) added to cover contribution
- [x] Documentation updated
- [x] Changelog updated
- [x] Coverage maintained or improved

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 95.45455% with 18 lines in your changes missing coverage. Please review.

Project coverage is 95.92%. Comparing base (1e35888) to head (f66f8e2).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #551 +/- ## ========================================== + Coverage 95.30% 95.92% +0.61% ========================================== Files 25 26 +1 Lines 3664 3899 +235 Branches 803 768 -35 ========================================== + Hits 3492 3740 +248 + Misses 83 69 -14 - Partials 89 90 +1 ``` | [Files](https://app.codecov.io/gh/calliope-project/calliope/pull/551?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=calliope-project) | Coverage Δ | | |---|---|---| | [src/calliope/backend/\_\_init\_\_.py](https://app.codecov.io/gh/calliope-project/calliope/pull/551?src=pr&el=tree&filepath=src%2Fcalliope%2Fbackend%2F__init__.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=calliope-project#diff-c3JjL2NhbGxpb3BlL2JhY2tlbmQvX19pbml0X18ucHk=) | `100.00% <100.00%> (+12.50%)` | :arrow_up: | | [src/calliope/backend/latex\_backend\_model.py](https://app.codecov.io/gh/calliope-project/calliope/pull/551?src=pr&el=tree&filepath=src%2Fcalliope%2Fbackend%2Flatex_backend_model.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=calliope-project#diff-c3JjL2NhbGxpb3BlL2JhY2tlbmQvbGF0ZXhfYmFja2VuZF9tb2RlbC5weQ==) | `96.26% <100.00%> (ø)` | | | [src/calliope/backend/parsing.py](https://app.codecov.io/gh/calliope-project/calliope/pull/551?src=pr&el=tree&filepath=src%2Fcalliope%2Fbackend%2Fparsing.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=calliope-project#diff-c3JjL2NhbGxpb3BlL2JhY2tlbmQvcGFyc2luZy5weQ==) | `96.88% <100.00%> (+0.85%)` | :arrow_up: | | [src/calliope/model.py](https://app.codecov.io/gh/calliope-project/calliope/pull/551?src=pr&el=tree&filepath=src%2Fcalliope%2Fmodel.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=calliope-project#diff-c3JjL2NhbGxpb3BlL21vZGVsLnB5) | `96.29% <ø> (+1.55%)` | :arrow_up: | | [src/calliope/backend/expression\_parser.py](https://app.codecov.io/gh/calliope-project/calliope/pull/551?src=pr&el=tree&filepath=src%2Fcalliope%2Fbackend%2Fexpression_parser.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=calliope-project#diff-c3JjL2NhbGxpb3BlL2JhY2tlbmQvZXhwcmVzc2lvbl9wYXJzZXIucHk=) | `94.00% <95.23%> (+0.46%)` | :arrow_up: | | [src/calliope/backend/pyomo\_backend\_model.py](https://app.codecov.io/gh/calliope-project/calliope/pull/551?src=pr&el=tree&filepath=src%2Fcalliope%2Fbackend%2Fpyomo_backend_model.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=calliope-project#diff-c3JjL2NhbGxpb3BlL2JhY2tlbmQvcHlvbW9fYmFja2VuZF9tb2RlbC5weQ==) | `98.07% <97.29%> (+2.46%)` | :arrow_up: | | [src/calliope/backend/backend\_model.py](https://app.codecov.io/gh/calliope-project/calliope/pull/551?src=pr&el=tree&filepath=src%2Fcalliope%2Fbackend%2Fbackend_model.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=calliope-project#diff-c3JjL2NhbGxpb3BlL2JhY2tlbmQvYmFja2VuZF9tb2RlbC5weQ==) | `97.66% <90.90%> (-0.43%)` | :arrow_down: | | [src/calliope/backend/gurobi\_backend\_model.py](https://app.codecov.io/gh/calliope-project/calliope/pull/551?src=pr&el=tree&filepath=src%2Fcalliope%2Fbackend%2Fgurobi_backend_model.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=calliope-project#diff-c3JjL2NhbGxpb3BlL2JhY2tlbmQvZ3Vyb2JpX2JhY2tlbmRfbW9kZWwucHk=) | `95.60% <95.60%> (ø)` | | ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/calliope-project/calliope/pull/551/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=calliope-project)

[brynpickering]

I have added tests, but we don't have gurobipy installed in the CI builds so those tests are being skipped. Hence the massive coverage drop.

[brynpickering]

I have run the urban scale model for a full year using the pyomo (with gurobi solver) and gurobi interfaces and get the attached results.

Headline: Gurobi interface peaks at ~30% less memory than pyomo. I know you got different results when running locally @sjpfenninger. This could be because I'm using a cloud license for Gurobi which I think means it offloads the actual solving onto a remote machine.

memray_flamegraphs.zip

---

gurobi run:

import calliope

calliope.set_log_verbosity("info")
m = calliope.examples.urban_scale(time_subset=None)
m.build(backend="gurobi")
m.solve(solver_options={"Threads": 1})

pyomo run:

import calliope

calliope.set_log_verbosity("info")
m = calliope.examples.urban_scale(time_subset=None)
m.build(backend="pyomo")
m.solve(
solver="gurobi", solver_options={"Threads": 1})

Profiling using memray and setting [out] and [runfile] separately for pyomo and gurobi runs:

memray run --trace-python-allocators -o [out].bin [runfile].py    
memray flamegraph [out].bin -o [out].html    

[brynpickering]

Last week's new pyomo release has caused an issue in one of our tests (see #597). I've pinned Pyomo upper bound to avoid this issue for now.

[brynpickering]

Thanks for the review @irm-codebase - updated!

[brynpickering]

We can't provide generalised instructions for cbc in the installation guide due to Windows users not being able to install it directly (and pixi wouldn't solve that). I would rather move to using HiGHs as our default open source solver, but I haven't got it to work with the pyomo kernel library...