The annual growth data should be compared with the base of the same year.
But diff() only will make it to compare the data with the next year.
shift(-1) will fix this problem.
below are some tests to show the difference between diff() and diff().shift(-1)
diff() without shift:
abs_diffs:
Time
1997.0 NaN <= not good
1998.0 96.0 <= this should be 1997 growth
1999.0 1402.0
2000.0 -2404.0
… the next year.
The annual growth data should be compared with the base of the same year. But diff() only will make it to compare the data with the next year. shift(-1) will fix this problem.
below are some tests to show the difference between diff() and diff().shift(-1)
diff() without shift:
abs_diffs: Time 1997.0 NaN <= not good 1998.0 96.0 <= this should be 1997 growth 1999.0 1402.0 2000.0 -2404.0
rel_diffs numbers: 1997.0 NaN <= 1998.0 0.033743 <= 1999.0 0.330115 2000.0 -1.304395 <=
shift data one row up:
pop_series.diff().shift(-1)
Time 1997.0 96.0 <= this is 1997 growth 1998.0 1402.0 <= 1999.0 -2404.0 2000.0 719.0 2001.0 -788.0
rel_diffs as below: Time 1997.0 0.034922 <= 1998.0 0.492794 <= 1999.0 -0.566047 2000.0 0.390125 <=