transportenergy / database

Tools for accessing and maintaining the iTEM model & historical databases
https://transportenergy.rtfd.io
GNU General Public License v3.0
24 stars 8 forks source link

T003: Missing data related to the mode "Road" and mode "Total Inland FT" #61

Open hlinero opened 3 years ago

hlinero commented 3 years ago

Issue regarding https://stats.oecd.org/Index.aspx?DataSetCode=ITF_GOODS_TRANSPORT#

Rationale 1: Mode "Road" and Vehicle Type (VT) "All" is broken down into two VTs: "Road freight transport for hire and reward" and "Road freight transport on own account." There are years in which the "All" VT is presented but one if its components is missing.

Rationale 2: Just as mentioned before, the "Inland" mode has "All" as VT. However, the components that make up the "All" include the VT "Pipeline," "Inland Waterway," "Rail," and "Road ALL". There are years in which the "All" VT is presented but one if its components is missing.

Below is the list of countries Albania:

Belarus:

Bosnia-Herzegovina:

Bulgaria:

Canada:

China:

Croatia

Czech Republic:

Denmark:

Greece:

India:

Ireland:

Japan

Korea:

Latvia:

Liechtenstein:

Serbia, Republic of:

Spain:

Switzerland:

Ukraine:

United Kingdom:

United States:

Currently, the issue is handled in the code https://github.com/transportenergy/database/blob/master/item/historical/scripts/T003.py

The procedures to fix this problem are the following:

  1. ITF correct these errors in their data and republish correct data https://stats.oecd.org/Index.aspx?DataSetCode=ITF_GOODS_TRANSPORT#
  2. Implement code change https://github.com/transportenergy/database/blob/master/item/historical/scripts/T003.py removing the cleaning scripts associated with this problem
  3. Regenerate a new merged dataset