Migrate MESSAGEix-Materials updates

macflo8 commented 2 months ago

This PR migrates the remaining updates of MESSAGEix-Materials to message-ix-models and is planned to complement #130 and potentially supersede it eventually.

Major model changes

Demand projections for all materials commodities now processed all in one module

Base year demand data is moved from code to separate yaml files (example)

Implementation of SSP differentiated dimensions:

Demand projections depend on population, GDP and elasticity assumptions of respective SSP
Cost projections can be updated to SSP assumptions utilizing the message-ix-models costs tool

Refactored base year harmonization of industry energy consumption using latest IEA data

Steps run so far

The code has been migrated from the private repository with the procedure described in the documentation.

Further steps

Write tests
Compare with #130 and make sure all updates are collected in this branch

How to review

Read the diff and note that the CI checks all pass.
Run a specific code snippet or command and check the output.
Build the documentation and look at a certain page.
Ensure that changes/additions are self-documenting, i.e. that another developer (someone like the reviewer) will be able to understand what the code does in the future.

PR checklist

~~- [ ] Continuous integration checks all ✅~~ Not possible due to missing tests

~- Add or expand tests; coverage checks both ✅~ See https://github.com/iiasa/message-ix-models/pull/189#issuecomment-2128709498; will be handled via #194
[x] Add, expand, or update documentation.
[x] Update doc/whatsnew.

codecov[bot] commented 2 months ago

Codecov Report

Attention: Patch coverage is 9.71330% with 3653 lines in your changes are missing coverage. Please review.

Project coverage is 52.2%. Comparing base (31950ed) to head (f214ed0).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #188 +/- ## ======================================== - Coverage 76.7% 52.2% -24.6% ======================================== Files 112 141 +29 Lines 7154 11342 +4188 ======================================== + Hits 5491 5921 +430 - Misses 1663 5421 +3758 ``` | [Files](https://app.codecov.io/gh/iiasa/message-ix-models/pull/188?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa) | Coverage Δ | | |---|---|---| | [message\_ix\_models/cli.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/188?src=pr&el=tree&filepath=message_ix_models%2Fcli.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvY2xpLnB5) | `93.4% <ø> (ø)` | | | [...age\_ix\_models/util/compat/message\_data/\_\_init\_\_.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/188?src=pr&el=tree&filepath=message_ix_models%2Futil%2Fcompat%2Fmessage_data%2F__init__.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdXRpbC9jb21wYXQvbWVzc2FnZV9kYXRhL19faW5pdF9fLnB5) | `100.0% <100.0%> (ø)` | | | [...s/util/compat/message\_data/get\_historical\_years.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/188?src=pr&el=tree&filepath=message_ix_models%2Futil%2Fcompat%2Fmessage_data%2Fget_historical_years.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdXRpbC9jb21wYXQvbWVzc2FnZV9kYXRhL2dldF9oaXN0b3JpY2FsX3llYXJzLnB5) | `20.0% <ø> (ø)` | | | [...ge\_ix\_models/util/compat/message\_data/get\_nodes.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/188?src=pr&el=tree&filepath=message_ix_models%2Futil%2Fcompat%2Fmessage_data%2Fget_nodes.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdXRpbC9jb21wYXQvbWVzc2FnZV9kYXRhL2dldF9ub2Rlcy5weQ==) | `100.0% <ø> (ø)` | | | [...util/compat/message\_data/get\_optimization\_years.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/188?src=pr&el=tree&filepath=message_ix_models%2Futil%2Fcompat%2Fmessage_data%2Fget_optimization_years.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdXRpbC9jb21wYXQvbWVzc2FnZV9kYXRhL2dldF9vcHRpbWl6YXRpb25feWVhcnMucHk=) | `28.5% <ø> (ø)` | | | [...ge\_ix\_models/util/compat/message\_data/utilities.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/188?src=pr&el=tree&filepath=message_ix_models%2Futil%2Fcompat%2Fmessage_data%2Futilities.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdXRpbC9jb21wYXQvbWVzc2FnZV9kYXRhL3V0aWxpdGllcy5weQ==) | `25.0% <ø> (ø)` | | | [...els/util/compat/message\_data/update\_h2\_blending.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/188?src=pr&el=tree&filepath=message_ix_models%2Futil%2Fcompat%2Fmessage_data%2Fupdate_h2_blending.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdXRpbC9jb21wYXQvbWVzc2FnZV9kYXRhL3VwZGF0ZV9oMl9ibGVuZGluZy5weQ==) | `11.7% <11.7%> (ø)` | | | [...til/compat/message\_data/update\_fix\_and\_inv\_cost.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/188?src=pr&el=tree&filepath=message_ix_models%2Futil%2Fcompat%2Fmessage_data%2Fupdate_fix_and_inv_cost.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdXRpbC9jb21wYXQvbWVzc2FnZV9kYXRhL3VwZGF0ZV9maXhfYW5kX2ludl9jb3N0LnB5) | `13.9% <13.9%> (ø)` | | | [message\_ix\_models/model/material/build.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/188?src=pr&el=tree&filepath=message_ix_models%2Fmodel%2Fmaterial%2Fbuild.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvbW9kZWwvbWF0ZXJpYWwvYnVpbGQucHk=) | `17.1% <17.1%> (ø)` | | | [message\_ix\_models/model/material/data\_generic.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/188?src=pr&el=tree&filepath=message_ix_models%2Fmodel%2Fmaterial%2Fdata_generic.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvbW9kZWwvbWF0ZXJpYWwvZGF0YV9nZW5lcmljLnB5) | `15.1% <15.1%> (ø)` | | | ... and [20 more](https://app.codecov.io/gh/iiasa/message-ix-models/pull/188?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa) | |

glatterf42 commented 2 months ago

Thanks @macflo8, this will be very useful :) Two immediate questions, though, about the files included: a lot of them are in message_ix_models/data/material/version_control/ and I'm wondering: is version_control really a subtopic of material? If not, I suspect this directory was created to distinguish files under version control and those without, but all files on GitHub are necessarily under version control, so there would be no need to keep this. Something like git mv message_ix_models/data/material/version_control/* message_ix_models/data/material/ might be enough to move all files up on level. Second, some files are in .../material/deprecated/, which again makes me wonder: if these files are deprecated, do we really need to keep them around?

khaeru commented 2 months ago

Something like git mv message_ix_models/data/material/version_control/* message_ix_models/data/material/ might be enough to move all files up on[e] level.

This could also be accomplished by giving a --path-rename argument to git filter-repo during the filtering process that moves files in that specific way.

khaeru commented 2 months ago

At https://github.com/iiasa/message-ix-models/pull/189#issuecomment-2078748183 I mentioned the commit message format in our code style.

For this longer branch, it seems there are a lot more commits affected, so maybe the suggestion of manual changes during git rebase -i is not realistic. Please have a look at https://github.com/iiasa/message-ix-models/pull/107#issuecomment-2062609910, particularly the replacements.txt file used; I think extending this file with an appropriate regex would be an easy way to automatically rewrite e.g. "Feat(materials): make petrochemical elasticity ssp dependent" to "Make petrochemical elasticity SSP dependent".

Second, note that #186 has been merged recently. While it is not necessary to constantly rebase this branch and the one for #189, both should be based on the latest main at the future moment when we eventually actually do the two merges.

macflo8 commented 2 months ago

Thanks @macflo8, this will be very useful :) Two immediate questions, though, about the files included: a lot of them are in message_ix_models/data/material/version_control/ and I'm wondering: is version_control really a subtopic of material? If not, I suspect this directory was created to distinguish files under version control and those without, but all files on GitHub are necessarily under version control, so there would be no need to keep this. Something like git mv message_ix_models/data/material/version_control/* message_ix_models/data/material/ might be enough to move all files up on level. Second, some files are in .../material/deprecated/, which again makes me wonder: if these files are deprecated, do we really need to keep them around?

Thanks for the comments @glatterf42

ad 1) Since we still rely on xlsx files for input data in MESSAGEix-Materials, keeping track of changes in these files is challenging since the changes are not displayed in human readable form. As a workaround I have added a simple CLI command to create a copies of all sheets in our xlsx files in csv format. Since we need to merge changes from some other branches like this one, the csv files can help to identify the changes in the input files. So these files are not required for the MESSAGEix-Materials model and can be deleted before merging to main. This is not a very elegant solution, so if you have ideas for a better solution I am happy to discuss. My preference would be to move away from xlsx files eventually. But this would require a significant effort and a refactoring of input data is not at the top of our priority list at the moment..

ad 2) that is a good catch. Yes, I think they can be deleted. I see that they are also still present on the original migrate branch. I will discuss it with @GamzeUnlu95.

glatterf42 commented 1 month ago

Thanks for the explanation, @macflo8 :) For the input data file format, I don't have any good solution at the top of my head. I can however tell you that the water and buildings modules use csv files for their input data, as far as I know. So maybe what you're doing is close to what you need to be compatible with them already: convert all excel files to csv once and adapt your code to accept that format rather than xlsx. I understand if this is not a top priority; there's no need to do this work in this PR (it might even be considered scope creep). We can always open clean-up PRs with specific tasks to keep reviews manageable :)

glatterf42 commented 1 month ago

This should be celebrated and mentioned in the next weekly MESSAGE meeting (at least :)

macflo8 commented 1 month ago

Yes definitely! 🥳 Many thanks to both of you @glatterf42 @khaeru for the help and guidance with the migration and everything else for this PR!

khaeru commented 1 month ago

Very well done, and congrats!

iiasa / message-ix-models