Fix discovered issues and improve MESSAGEix-Materials

[macflo8]

• Removes message_data dependencies (some .yaml and .csv input files).
• Fixes incorrect imports of previously migrated code in utils/compat/message_data.
• Removes deprecated/unused variables and code sections.
• Adresses #198.
• Implements data packaging mentioned in https://github.com/iiasa/message-ix-models/issues/194#issuecomment-2129269295.
• Hide some debugging and project specific CLI commands of material-ix group
• Moves from using untested helpers in utils/compat/message_data to using tested equivalents of message-ix-models
• Removes click options from material-ix commands that are already implemented in message_ix_models/cli.py
• Stores csv files in data/material as text (previously stored with git LFS)
• General updates to documentation

Details: Building MESSAGEix-Materials on an existing MESSAGEix-GLOBIOM scenario still requires a few external data file that have previously been stored in a private repository. The respective files are migrated to message-ix-models/data and data paths in the modules updated accordingly in this PR. The files located in util/compat/message_data migrated in the course of #188 contain some incorrect import statements, which are fixed in this PR.

The following files are needed in the build and were therefore added:

• SP_UE_dyn_input.xlsx
• UE_dyn_input_all.xlsx
• IEA_mappings.csv
• IEA_mappings_chemicals.csv
• IEA_mappings_industry.csv

The build however still requires proprietary data from IEA Extended Energy Balances to calibrate historical industry activity. The path to the required file can be specified with a newly added CLI option --iea_data_path in the build command.

All input data for MESSAGEix-Materials is now packaged in a .tar.gz file, that can be fetched with mix-models fetch MESSAGEix-Materials.

When executing mix-models material-ix --help only the basic comments related to building/solving/reporting are shown

How to review

Run build/solve/report with material-ix CLI commands and confirm correct execution of each task.

PR checklist

• [ ] Continuous integration checks all ✅ Not possible due to missing tests
• [ ] Add or expand tests; coverage checks both ✅ will be handled via https://github.com/iiasa/message-ix-models/issues/194
• [x] Add, expand, or update documentation.
• [x] Update doc/whatsnew.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 29.32166% with 323 lines in your changes missing coverage. Please review.

Project coverage is 51.9%. Comparing base (229eecb) to head (4814e7a).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #201 +/- ## ======================================= - Coverage 52.2% 51.9% -0.3% ======================================= Files 141 142 +1 Lines 11346 11250 -96 ======================================= - Hits 5925 5843 -82 + Misses 5421 5407 -14 ``` | [Files](https://app.codecov.io/gh/iiasa/message-ix-models/pull/201?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa) | Coverage Δ | | |---|---|---| | [message\_ix\_models/cli.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/201?src=pr&el=tree&filepath=message_ix_models%2Fcli.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvY2xpLnB5) | `93.4% <ø> (ø)` | | | [...ssage\_ix\_models/model/material/data\_ammonia\_new.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/201?src=pr&el=tree&filepath=message_ix_models%2Fmodel%2Fmaterial%2Fdata_ammonia_new.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvbW9kZWwvbWF0ZXJpYWwvZGF0YV9hbW1vbmlhX25ldy5weQ==) | `8.7% <100.0%> (+0.2%)` | :arrow_up: | | [...l/material/material\_demand/material\_demand\_calc.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/201?src=pr&el=tree&filepath=message_ix_models%2Fmodel%2Fmaterial%2Fmaterial_demand%2Fmaterial_demand_calc.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvbW9kZWwvbWF0ZXJpYWwvbWF0ZXJpYWxfZGVtYW5kL21hdGVyaWFsX2RlbWFuZF9jYWxjLnB5) | `17.3% <ø> (-1.6%)` | :arrow_down: | | [...e\_ix\_models/tests/model/material/test\_data\_util.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/201?src=pr&el=tree&filepath=message_ix_models%2Ftests%2Fmodel%2Fmaterial%2Ftest_data_util.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdGVzdHMvbW9kZWwvbWF0ZXJpYWwvdGVzdF9kYXRhX3V0aWwucHk=) | `100.0% <100.0%> (ø)` | | | [...s/util/compat/message\_data/get\_historical\_years.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/201?src=pr&el=tree&filepath=message_ix_models%2Futil%2Fcompat%2Fmessage_data%2Fget_historical_years.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdXRpbC9jb21wYXQvbWVzc2FnZV9kYXRhL2dldF9oaXN0b3JpY2FsX3llYXJzLnB5) | `20.0% <ø> (ø)` | | | [...util/compat/message\_data/get\_optimization\_years.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/201?src=pr&el=tree&filepath=message_ix_models%2Futil%2Fcompat%2Fmessage_data%2Fget_optimization_years.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdXRpbC9jb21wYXQvbWVzc2FnZV9kYXRhL2dldF9vcHRpbWl6YXRpb25feWVhcnMucHk=) | `28.5% <ø> (ø)` | | | [message\_ix\_models/util/pooch.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/201?src=pr&el=tree&filepath=message_ix_models%2Futil%2Fpooch.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdXRpbC9wb29jaC5weQ==) | `42.0% <ø> (ø)` | | | [.../compat/message\_data/change\_technology\_lifetime.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/201?src=pr&el=tree&filepath=message_ix_models%2Futil%2Fcompat%2Fmessage_data%2Fchange_technology_lifetime.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdXRpbC9jb21wYXQvbWVzc2FnZV9kYXRhL2NoYW5nZV90ZWNobm9sb2d5X2xpZmV0aW1lLnB5) | `2.5% <0.0%> (ø)` | | | [...at/message\_data/check\_scenario\_fix\_and\_inv\_cost.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/201?src=pr&el=tree&filepath=message_ix_models%2Futil%2Fcompat%2Fmessage_data%2Fcheck_scenario_fix_and_inv_cost.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdXRpbC9jb21wYXQvbWVzc2FnZV9kYXRhL2NoZWNrX3NjZW5hcmlvX2ZpeF9hbmRfaW52X2Nvc3QucHk=) | `6.2% <50.0%> (ø)` | | | [...els/util/compat/message\_data/update\_h2\_blending.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/201?src=pr&el=tree&filepath=message_ix_models%2Futil%2Fcompat%2Fmessage_data%2Fupdate_h2_blending.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdXRpbC9jb21wYXQvbWVzc2FnZV9kYXRhL3VwZGF0ZV9oMl9ibGVuZGluZy5weQ==) | `18.7% <66.6%> (+6.9%)` | :arrow_up: | | ... and [9 more](https://app.codecov.io/gh/iiasa/message-ix-models/pull/201?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa) | | ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/iiasa/message-ix-models/pull/201/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa)

[macflo8]

* Files like message_ix_models/data/material/iea_mappings/*.csv:
  * should be stored as text, not using Git LFS.
  * contain the same phrase "iea_mappings" in their path and filename; this could be simplified.
  * do not appear to be mentioned anywhere in the documentation.

At the moment our biggest csv file is ~400 KB. Is it okay to store all of them as text?

* `message_data.tools.utilities.get_optimization_years()` does the same thing as [`ScenarioInfo.Y`](https://docs.messageix.org/projects/models/en/latest/api/util.html#message_ix_models.util.scenarioinfo.ScenarioInfo.Y). The latter is computed only once, and has tests and documentation. The former is not tested, and hits the database every time it is called. I would prefer that we take the opportunity, when moving code to `message_ix_models`, to use the better-maintained solution. Similar applies to `get_historical_years()`.

Just to be clear: This means we should refactor the functions that use these helpers and pass a ScenarioInfo instance to them instead, correct?

[khaeru]

At the moment our biggest csv file is ~400 KB. Is it okay to store all of them as text?

Generally yes. Git will automatically compress these files. Some further nuances:

• If the file will be manually adjusted or edited, then it is helpful to see those changes as a diff of a few lines. When stored as plain text, Git can provide this.
• If the file is wholesale output of another program and is clearly documented ("The file foobar.csv comes from running foo bar --option=X"), and will only ever be entirely replaced, then diffs would not be needed or very useful, and Git LFS can be used.

Just to be clear: This means we should refactor the functions that use these helpers and pass a ScenarioInfo instance to them instead, correct?

• Only for files that are added or modified in this PR.
• It's not strictly necessary to entirely refactor or change the signature of the calling function. It's also possible as an intermediate step to replace e.g. https://github.com/iiasa/message-ix-models/blob/c8319130eab17335c4bbc8bf0eb1b9aecf92fb9d/message_ix_models/util/compat/message_data/manual_updates_ENGAGE_SSP2_v417_to_v418.py#L419 with something like:

  model_years = ScenarioInfo(scen).Y

  This is already an improvement; it will later make it easier to see where the ScenarioInfo object can be passed/used instead of the scenario itself. Since we may want to provide tested and documented alternatives for the codes in .util.compat.message_data, let's not invest too much time in tinkering with them.

[macflo8]

Okay thanks for confirming! I pushed the suggested changes. I think all the requested changes and comments are addressed now.