iiasa / message-ix-models

Tools for the MESSAGEix-GLOBIOM family of models
https://docs.messageix.org/models
Apache License 2.0
17 stars 33 forks source link

Convert Intratec data from Excel to CSV #167

Closed measrainsey closed 5 months ago

measrainsey commented 5 months ago

Convert Intratec data from Excel spreadsheet to CSV file

How to review

Read the diff and note that the CI checks all pass.

PR checklist

codecov[bot] commented 5 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 76.7%. Comparing base (8f39459) to head (c62cbf5). Report is 3555 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #167 +/- ## ======================================= - Coverage 78.0% 76.7% -1.3% ======================================= Files 112 112 Lines 7155 7154 -1 ======================================= - Hits 5585 5494 -91 - Misses 1570 1660 +90 ``` | [Files with missing lines](https://app.codecov.io/gh/iiasa/message-ix-models/pull/167?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa) | Coverage Δ | | |---|---|---| | [...tests/tools/costs/test\_regional\_differentiation.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/167?src=pr&el=tree&filepath=message_ix_models%2Ftests%2Ftools%2Fcosts%2Ftest_regional_differentiation.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdGVzdHMvdG9vbHMvY29zdHMvdGVzdF9yZWdpb25hbF9kaWZmZXJlbnRpYXRpb24ucHk=) | `100.0% <ø> (ø)` | | | [...\_ix\_models/tools/costs/regional\_differentiation.py](https://app.codecov.io/gh/iiasa/message-ix-models/pull/167?src=pr&el=tree&filepath=message_ix_models%2Ftools%2Fcosts%2Fregional_differentiation.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa#diff-bWVzc2FnZV9peF9tb2RlbHMvdG9vbHMvY29zdHMvcmVnaW9uYWxfZGlmZmVyZW50aWF0aW9uLnB5) | `97.9% <100.0%> (-0.1%)` | :arrow_down: | ... and [5 files with indirect coverage changes](https://app.codecov.io/gh/iiasa/message-ix-models/pull/167/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa)
measrainsey commented 5 months ago

Looks good to me :)

You might want to consider the name of your new file, though. When referring to regions, I think we typically use an upper-case R, so R12 in this case. Also, if you foresee adding indices for other types of region distributions and possibly more files specific to them, you could also make R12 a folder within data/intratec/ and just have a file called indices.csv in there.

Thanks @glatterf42! I implemented your advice and renamed the file/reorganized the folder structure.

I also added a line about the PR in doc/whatsnew.

khaeru commented 5 months ago

@measrainsey, looking at where your code uses this data, I see this:

https://github.com/iiasa/message-ix-models/blob/8f39459fa2e978f80295d17feb0b052909dbeac2/message_ix_models/tools/costs/regional_differentiation.py#L560-L575

So it is not clear whether the file is for the R12 node codelist, or for R11 and gets adapted. From the values it seems the latter.

I suggest:

measrainsey commented 5 months ago

I suggest:

* Store the data for R11. Include the exact node names, like `R11_NAM`, in a column called "node".

* Call the existing function [`adapt_R11_R12`](https://docs.messageix.org/projects/models/en/latest/api/util.html#message_ix_models.util.node.adapt_R11_R12).

@khaeru: As per your suggestion, I changed the indices file to be for R11 instead (putting that file in an R11 folder), and now use adapt_R11_R12 when the R12 node is activated. (Just a note for myself on here that in order for adapt_R11_R12 to work, the other column in the DataFrame that is passed through had to be called value.)

https://github.com/iiasa/message-ix-models/blob/c62cbf5df1728d3243415815185421c88ab11852/message_ix_models/tools/costs/regional_differentiation.py#L539-L555