capital / consumption jumps between Nov / Dec 2022

orichters commented 1 year ago

FYI @robertpietzcker, @mleimbach, @LaviniaBaumstark, @aodenweller, @fschreyer.

End of November/beginning of December 2022, the calibration changed substantially, affecting consumption, investment and capital stock behavior.

Capital stocks increased:

Consumption shows a significant drop after 2005:

Investment is higher:

Somehow, the CES parameters changed substantially, for example in REF, but also in EUR, as extracted with the small script attached. The relevant AMT runs in /p/projects/remind/modeltests/output/ are:

SSP2EU-AMT-Base_2022-11-26_00.08.52 vs. SSP2EU-AMT-Base_2022-12-03_00.12.29
SSP2EU-AMT-calibrate_2022-11-26_00.07.54 vs. SSP2EU-AMT-calibrate_2022-12-03_00.10.49

Dear colleagues, it would be awesome if someone with a better intuition about what affects the calibration would have a look at this PR that I suspect may be causing the issue of consumption drop and investment spike in the 2010–2020 years, see attached document.

Somehow, the CES parameter changed substantially, for example in REF, but also in EUR, as extracted with the small script attached. The relevant runs are /p/projects/remind/modeltests/output/SSP2EU-AMT-Base_2022-11-26_00.08.52 and SSP2EU-AMT-Base_2022-12-03_00.12.29, respectively the SSP2EU-AMT-calibrate_2022-11-26_00.07.54 / SSP2EU-AMT-calibrate_2022-12-03_00.10.49 runs.

   scenario                   region period value all_in cesParameter
 1 SSP2EU-AMT-Base_2022-11-26 EUR      2005 0.540 lab    xi
 2 SSP2EU-AMT-Base_2022-12-03 EUR      2005 0.277 lab    xi
 3 SSP2EU-AMT-Base_2022-11-26 EUR      2010 0.540 lab    xi
 4 SSP2EU-AMT-Base_2022-12-03 EUR      2010 0.277 lab    xi
 5 SSP2EU-AMT-Base_2022-11-26 EUR      2005 0.235 kap    xi
 6 SSP2EU-AMT-Base_2022-12-03 EUR      2005 0.487 kap    xi
 7 SSP2EU-AMT-Base_2022-11-26 EUR      2010 0.235 kap    xi
 8 SSP2EU-AMT-Base_2022-12-03 EUR      2010 0.487 kap    xi
 9 SSP2EU-AMT-Base_2022-11-26 REF      2005 0.2   lab    xi
10 SSP2EU-AMT-Base_2022-12-03 REF      2005 0.2   lab    xi
11 SSP2EU-AMT-Base_2022-11-26 REF      2010 0.2   lab    xi
12 SSP2EU-AMT-Base_2022-12-03 REF      2010 0.2   lab    xi
13 SSP2EU-AMT-Base_2022-11-26 REF      2005 0.453 kap    xi
14 SSP2EU-AMT-Base_2022-12-03 REF      2005 0.558 kap    xi
15 SSP2EU-AMT-Base_2022-11-26 REF      2010 0.453 kap    xi
16 SSP2EU-AMT-Base_2022-12-03 REF      2010 0.558 kap    xi

A full compareScenarios2 is available at /p/tmp/oliverr/remind/compScen-consumption_issue-2023-03-31_13.11.07-H12.pdf.

Here is a comparison of all the changes between the SSP2EU-AMT-Base_2022-11-26 run and the 2022-12-03 run: https://github.com/remindmodel/remind/compare/81264e35..a3eb6ac7?w=1

My suspicion is this PR, but I'm not entirely sure.

mleimbach commented 1 year ago

Changes of this PR seem to be mostly energy system related. It would be surprising that this should cause such macro change. My suspect would be the change of the CES parameters (probably derived from a new calibration which itself could be include a number of different changes from previous PRs). Also the inputRevision number changes. Could there be an issue with new input data?

robertpietzcker commented 1 year ago

@mleimbach @orichters How are the 2005 non-ESM capital stocks determined in REMIND? are they an input data to which the variable is fixed?

mleimbach commented 1 year ago

I have to remind myself. So without guarantee: capital stocks are free, but more or less predetermined by the calibration which provides the (share and) efficiency parameters of capital. The calibration routine changed over the years. I am not sure how it works actually. But there is some capital stock specific information included: either a capital price that can be determined based on an assumed capital income share and a capital coefficient (K/Y) or a given capital stock trajectory that is based on given GDP and K/Y.

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented 1 year ago

How are the 2005 non-ESM capital stocks determined in REMIND? are they an input data to which the variable is fixed?

Non-ESM capital (everything in vm_cesIO(ppfKap) is an input into the calibration from file f29_capitalQuantity.cs4r. The variable is not fixed, but as it is a calibration target, it will pretty much hit the prescribed values by the time the calibration is done.

Calibration input:

$ diff -y <( dumpgdx SSP2EU-AMT-calibrate_2022-11-26_00.07.54/fulldata.gdx f29_capitalQuantity '20[01].,EUR,gdp_SSP2EU,kap[^a-z]' | grep -v '^2000' ) <( dumpgdx SSP2EU-AMT-calibrate_2022-12-03_00.10.49/fulldata.gdx f29_capitalQuantity '20[01].,EUR,gdp_SSP2EU,kap[^a-z]' | grep -v '^2000' )
2005  EUR  gdp_SSP2EU  kap                  45.675646        |  2005  EUR  gdp_SSP2EU  kap                  63.485514
2005  EUR  gdp_SSP2EU  kap_cement           0.003931         |  2005  EUR  gdp_SSP2EU  kap_cement           0.007092
2005  EUR  gdp_SSP2EU  kap_chemicals        0.009574         |  2005  EUR  gdp_SSP2EU  kap_chemicals        0.019769
2005  EUR  gdp_SSP2EU  kap_steel_primary    0.002434         |  2005  EUR  gdp_SSP2EU  kap_steel_primary    0.005806
2005  EUR  gdp_SSP2EU  kap_steel_secondary  0.00059      |  2005  EUR  gdp_SSP2EU  kap_steel_secondary  0.001134
2005  EUR  gdp_SSP2EU  kap_otherInd         2.220898         |  2005  EUR  gdp_SSP2EU  kap_otherInd         0.119269
2010  EUR  gdp_SSP2EU  kap                  48.12552         |  2010  EUR  gdp_SSP2EU  kap                  66.602287
2010  EUR  gdp_SSP2EU  kap_cement           0.004168         |  2010  EUR  gdp_SSP2EU  kap_cement           0.00659
2010  EUR  gdp_SSP2EU  kap_chemicals        0.011151         |  2010  EUR  gdp_SSP2EU  kap_chemicals        0.020252
2010  EUR  gdp_SSP2EU  kap_steel_primary    0.002324         |  2010  EUR  gdp_SSP2EU  kap_steel_primary    0.00584
2010  EUR  gdp_SSP2EU  kap_steel_secondary  0.000567         |  2010  EUR  gdp_SSP2EU  kap_steel_secondary  0.001129
2010  EUR  gdp_SSP2EU  kap_otherInd         2.619937         |  2010  EUR  gdp_SSP2EU  kap_otherInd         0.123344
2015  EUR  gdp_SSP2EU  kap                  50.775005        |  2015  EUR  gdp_SSP2EU  kap                  69.961441
2015  EUR  gdp_SSP2EU  kap_cement           0.004435         |  2015  EUR  gdp_SSP2EU  kap_cement           0.006569
2015  EUR  gdp_SSP2EU  kap_chemicals        0.013394         |  2015  EUR  gdp_SSP2EU  kap_chemicals        0.020857
2015  EUR  gdp_SSP2EU  kap_steel_primary    0.002143         |  2015  EUR  gdp_SSP2EU  kap_steel_primary    0.005829
2015  EUR  gdp_SSP2EU  kap_steel_secondary  0.000532         |  2015  EUR  gdp_SSP2EU  kap_steel_secondary  0.001088
2015  EUR  gdp_SSP2EU  kap_otherInd         3.183333         |  2015  EUR  gdp_SSP2EU  kap_otherInd         0.12639

Calibration output:

$ diff -y <( dumpgdx SSP2EU-AMT-calibrate_2022-11-26_00.07.54/fulldata.gdx vm_cesIO '20[01].,EUR,kap.*,L\>' ) <( dumpgdx SSP2EU-AMT-calibrate_2022-12-03_00.10.49/fulldata.gdx vm_cesIO '20[01].,EUR,kap.*,L\>' )
2005  EUR  kap                  L  45.9529912876245      |  2005  EUR  kap                  L  66.9989542470875
2005  EUR  kap_cement           L  0.00415857769748      |  2005  EUR  kap_cement           L  0.00750257772336
2005  EUR  kap_chemicals        L  0.01012826834792      |  2005  EUR  kap_chemicals        L  0.02091348829852
2005  EUR  kap_steel_primary    L  0.00257491175672      |  2005  EUR  kap_steel_primary    L  0.00614212722248
2005  EUR  kap_steel_secondary  L  0.0006241569172       |  2005  EUR  kap_steel_secondary  L  0.00119965075272
2005  EUR  kap_otherInd         L  2.34947262558584      |  2005  EUR  kap_otherInd         L  0.12617384975852
2010  EUR  kap                  L  48.5129788171281      |  2010  EUR  kap                  L  69.3227091058502
2010  EUR  kap_cement           L  0.00428123645226762       |  2010  EUR  kap_cement           L  0.00705534999191206
2010  EUR  kap_chemicals        L  0.0115592347538713        |  2010  EUR  kap_chemicals        L  0.0208112019814236
2010  EUR  kap_steel_primary    L  0.00244786506330071       |  2010  EUR  kap_steel_primary    L  0.00610218654641758
2010  EUR  kap_steel_secondary  L  0.000542297248669287      |  2010  EUR  kap_steel_secondary  L  0.001116553251286
2010  EUR  kap_otherInd         L  2.71558624187911      |  2010  EUR  kap_otherInd         L  0.125625986496384
2015  EUR  kap                  L  50.3772977626294      |  2015  EUR  kap                  L  74.3373490709143
2015  EUR  kap_cement           L  0.00476266340122844       |  2015  EUR  kap_cement           L  0.00723074412570194
2015  EUR  kap_chemicals        L  0.0146241479955457        |  2015  EUR  kap_chemicals        L  0.0224945594237425
2015  EUR  kap_steel_primary    L  0.00238542485567444       |  2015  EUR  kap_steel_primary    L  0.00643101382013324
2015  EUR  kap_steel_secondary  L  0.000595162919914336      |  2015  EUR  kap_steel_secondary  L  0.00120207297749892
2015  EUR  kap_otherInd         L  3.52363935964865      |  2015  EUR  kap_otherInd         L  0.135921445082906

So that changed between

$ ag mrremind rev6.316_62eff8f7_remind/f29_capitalQuantity.cs4r rev6.324_62eff8f7_remind/f29_capitalQuantity.cs4r 
rev6.316_62eff8f7_remind/f29_capitalQuantity.cs4r
3:* origin: calcOutput(type = "Capital", file = "f29_capitalQuantity.cs4r", round = 6) (madrat 2.19.3 | mrremind 0.141.1)

rev6.324_62eff8f7_remind/f29_capitalQuantity.cs4r
3:* origin: calcOutput(type = "Capital", file = "f29_capitalQuantity.cs4r", round = 6) (madrat 2.20.6 | mrremind 0.147.1)

pik-piam/mrremind#307 and pik-piam/mrremind#328.

That makes for 18 merge requests into mrremind that could be responsible:

$ echo -e 'hash|merge request|version|message\n---|---|---|---'; for h in $( git log --format='%h' --abbrev-commit 1242136^1..298465a --grep "Merge pull request" ); do v=$( git show "$h":./DESCRIPTION | sed -n 's/Version: //p' ); git --no-pager log -1 --format='%h;%s;%b' "$h" | sed "s/;Merge pull request #\([0-9]*\).*;/ | [\1](https:\/\/github.com\/pik-piam\/mrremind\/pull\/\1) | $v |/"; done

hash	merge request	version	message
298465a	328	0.147.1	Update EEA Projections
dd270f5	327	0.147.0	add new data source: Ember
a022059	321	0.146.0	cement production matches historic data
da0a7cc	326	0.145.4	adjust weights in 'calcEmissionFactorsFeedstocks' to avoid zeros in a…
57fc7ad	323	0.145.4	bug fix
2d20da2	320	0.145.3	Avoid duplication of scenarios due to imprecise filtering.
8a69d64	322	0.145.3	fix error in readTCdamageKrichene
7cbdd29	319	0.145.3	Corrections for tropical cyclone input data now not dependent on SSP …
e0a8968	318	0.145.2	Include SE Electricity variables from AGEB in historical.mif
fe97a4f	317	0.145.1	clean various roxygen warnings
d7bab45	306	0.145.0	integrate industry EEK calculation, update cement production source
9b026c1	316	0.144.0	SSP2_lowEn for all standard scenario combinations
176c14b	315	0.143.1	bump UNFCCC to 2022 data
e48a8ef	313	0.143.0	Calculate emission factors for feedstocks in the chemicals industry
d06a453	311	0.142.0	add calcCement(), combining data from readvanRuijven2016() and readUSGS(cement)
847adc3	310	0.141.4	Correct mapping
073c0f3	309	0.141.3	Allow more scenario combinations
afc4939	308	0.141.2	Bugfix calc scenario combinations EDGE-Transport
1242136	307	0.141.1	I allow myself to merge this directly as it just fixes a typo.

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented 1 year ago

I guess I will look at pik-piam/mrremind#306 …

mleimbach commented 1 year ago

Thanks Michaja. Thus, it is quite likely that the jumps are caused by a change of capital stock input data with non-ESM capital showing the largest difference. Could anybody already detect in which of the considered PRs this change is hidden?

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented 1 year ago

Aaand we're fucked. I think we moved past pik-piam/mrremind#314, the switch to the new IEA data, which is kind of a point-of-no-return for testing older mrremind revisions. @fbenke-pik, can you confirm? If yes, what possibilities exist to run the 2016 IEA data?

fbenke-pik commented 1 year ago

We unfortunately don't have a switch to hop between old and new data, so I'd agree with the point-of-no-return phrasing. We could implement it if needed, but would involve extensive validation of the interplay of old data with all the libraries (as it affects many calc functions).

Other than that, I can only think of jumping back to the latest mrremind/mrcommons versions before the switch.

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented 1 year ago

Thus, it is quite likely that the jumps are caused by a change of capital stock input data with non-ESM capital showing the largest difference. Could anybody already detect in which of the considered PRs this change is hidden?

Well, pik-piam/mrremind#306 seems to be the only thing touching on the capital stocks, judging from the merge request messages (which are not always … comprehensive, shall we say?)

The energy efficiency capital (EEK) is subtracted from the general capital stock, to avoid double counting. There was a bug with too much EEK, but the jump in kap is way more than the decrease in EEK.

I'm not sure if something, and if so, what, in pik-piam/mrremind#306 could have caused the difference in kap.

But as it is questionable whether or when we will get a definitive answer on what caused the change, we should maybe think along different lines, too.

The capital stock is higher (Oliver's first plot), but the shape is the same.
Investments are higher, but the shape is the same, except for 2005 being the same in both input data revisions.
Consumption is lower, but the shape is the same, except for the 2010 dip, which seems to be caused by the 2005–10 jump in investments.

So, why did 2005 investments not change? If we are not married to the precise consumption path, only to the idea of an kink-free consumption path, having "smooth" investments from 2005 on might be the way to go.

mleimbach commented 1 year ago

I agree with Michaja's last proposal (smoothing 2005 investments). Investments in 2005 are currently fixed to an "empirical" value (which we may have to check). It probably does not fit to the new initial capital stock. HOWEVER, where does the change in the capital stock estimation comes from ???

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented 1 year ago

I agree with Michaja's last proposal (smoothing 2005 investments). Investments in 2005 are currently fixed to an "empirical" value (which we may have to check). It probably does not fit to the new initial capital stock.

Well, it's "real" world data (PWT investment shares). And it didn't really fit the the investments REMIND is doing before hand, but it did work out to a smooth behaviour.

HOWEVER, where does the change in the capital stock estimation comes from ???

Guess I will dissect that.

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented 1 year ago

Guess I will dissect that.

Before anything industry-related is happening, cap_macro[EUR,2005,'gdp_SSP2EU.kap'] is at 63.2. So it's not the EEK …

orichters commented 1 year ago

Is there anything I can do at the moment? Staring at the source code doesn't seem to work, and I miss the "how to debug input data errors" tutorial that would help me where to start :)

mleimbach commented 1 year ago

Michaja, thank you very much for digging deep and localizing the source of the problem. Now Macro has to do its homework.

orichters commented 1 year ago

fixed with https://github.com/pik-piam/mrdrivers/pull/67

remindmodel / remind

capital / consumption jumps between Nov / Dec 2022 #1276