T001: magnitude error in raw data for CHN

[noussan]

There seems to be an error in the T001 data for China, two orders of magnitude of difference for data up to 2001 and after 2002.

[khaeru]

@noussan thanks for looking closely and reporting this. @hlinero and I had a look and confirmed the issue.

Because this is a first instance of handling such reports from the community, I want to give an example of a complete, detailed description of both the issue and the fix. FYI @soniayeh.

A full description:

• This is a cached copy of the raw, input data for source T001. The data provider for T001 is the ITF-OECD: https://github.com/transportenergy/metadata/blob/be0183f/historical/sources.yaml#L12-L20
• The error affects the years 1990 to 2001 inclusive:
  • 1985: 523900 × 10⁶ tonne-kilometre ← this data seems to be in order, i.e. in the same units as 2002 and later
  • 1990: 8141 × 10⁶ tonne-kilometre ← erroneous data begins
  • …
  • 2001: 20873 × 10⁶ tonne-kilometre ← erroneous data ends
  • 2002: 523900 × 10⁶ tonne-kilometre
• The error does not appear to affect other regions. This indicates the error is probably present in the values given by the data source, in this instance probably the National Bureau of Statistics of China (NBS) or another, intermediate provider from which ITF-OECD obtains this data. (I have dealt with similar errors in NBS data in the past.)

I have transferred this issue from the transportenergy/metadata that contains the cached file, to the transportenergy/database repository, which contains the processing codes. The initial fix will be to modify these processing codes, specifically file T001.py, to:

1. In the check() function, perform an arithmetic check to confirm that the error exists. For instance, interpolate between the 1985 and 2002 data points, divide the results by the data points for 1990–2001; the values should be ~ 100, or >90, or similar.
2. In the process() function, adjust the data by multiplying by 100.

The longer-term fix is to notify the data provider (ITF-OECD) of the error in the data they are providing. They can then implement a fix, as & when they see fit.

The check (1) is critical because it will allow us to automatically detect when the error is, or is not, present in the input data. When the input data is corrected, then we can also remove the adjustment in the processing step (2). Without this check, the adjustment might be wrongly applied twice: first by the provider, and then again by the iTEM code, which would produce values that were erroneously too high by two orders of magnitude.

[khaeru]

There was a comment on the PR https://github.com/transportenergy/database/pull/40#issuecomment-772065295:

The fix was to multiply 100 to data 1985-2001. But I think the fix should be dividing 100 to data 2002 and after. Can you double check?

Here are a few rows from the input data:

"CHN","China","T-SEA-CAB","Coastal shipping (national transport)","1985","1985","TONNEKM","Tonnes-kilometres","6","Millions",,,532900,,
"CHN","China","T-SEA-CAB","Coastal shipping (national transport)","1990","1990","TONNEKM","Tonnes-kilometres","6","Millions",,,8141,,
[…]
"CHN","China","T-SEA-CAB","Coastal shipping (national transport)","2001","2001","TONNEKM","Tonnes-kilometres","6","Millions",,,20873,,
"CHN","China","T-SEA-CAB","Coastal shipping (national transport)","2002","2002","TONNEKM","Tonnes-kilometres","6","Millions",,,2173300,,
[…]
"CHN","China","T-SEA-CAB","Coastal shipping (national transport)","2017","2017","TONNEKM","Tonnes-kilometres","6","Millions",,,5508400,,

@soniayeh, to be clear, the interpretation (call it “A”) of the problem was that:

• The observation for 1985 is correct at 532900 million = 5.329 × 10¹¹ tonne-kilometre—likewise the observations from 2002 to 2017 inclusive.
• The observation for e.g. 1990 is not 8141 million = 8.141 × 10⁹ t km, but should be corrected to 8.141 × 10¹¹ t km.

This was based on what was in the notebooks that you and @hlinero developed together.

What you're suggesting is the opposite, interpretation (B):

• The observations from 1990 through 2001 are correct at e.g. 8.141×10⁹ t km.
• The observations up to 1985 inclusive and from 2002 onwards including all the latest observations, are all incorrect, and should be corrected downwards, e.g. from 5.329 × 10¹¹ t km to 5.329 × 10⁹ t km.

Here's another row:

"USA","United States","T-SEA-CAB","Coastal shipping (national transport)","2002","2002","TONNEKM","Tonnes-kilometres","6","Millions",,,384977,,

If this is correct, U.S. coastal shipping in 2002 was 3.85 × 10¹¹ t km. Then either:

• Interpretation (A): Chinese coastal shipping in the same year was 2173300 million = 2.17 × 10¹² t km, or roughly 5.6 times as much as in the U.S.
• Interpretation (B): Chinese coastal shipping in the same year was 2.17 × 10¹⁰ t km, or roughly 6% the value in the U.S.

[khaeru]

In Slack, @soniayeh replied:

Yes, I was thinking (B) should be more correct, but I am not sure. I wonder what the ITF colleagues think?

Over at #57, @RachelePoggi wrote:

If I understood correctly, T001 refers to coastal shipping. I think this issue has been fixed in our database. Maybe you can give a look and see if i looked at the right variable

and then @soniayeh wrote:

Yes. You are correct. The source (ITF) has corrected the data. Therefore [w]e should remove the temporary code fix in #40 to the script T001, without applying the multiplier of 100 for data up to 2001.

I can't understand from this exchange what "this issue" turned out to be actually be—was it (A) or (B)? What specifically was changed to fix it?