search

SWS-Methodology / faoswsTrade

World trade data processing for the FAO Statistical Working System
http://www.fao.org/economic/ess/ess-home/en/
5 stars 2 forks source link

Module sometimes defaults to mirror data even when official data is present #153

Closed baldwinkl closed 7 years ago

baldwinkl commented 7 years ago

In the last module run, we identified instances where the module substituted partner data for official data, although official data is present in the raw data.

One example of this is for Angola. The SWS datasets show that mirror data is currently being used for several partners across many products (Namibia and South Africa are the most prominent ones). However, Angola is in many cases reporting official trade flows for these partners (see raw data file), but they are for some reason overridden by the mirror flows.

Extraction from SWS: image

Raw data (24 is Angola, 710 is South Africa, 0701... is potatoes): image

In some cases, this was not a problem in the previous module runs, as evident from the history. See the below example on Australia's imports of beer from Mexico.

Extraction from SWS: image

Raw data (36 is Australia, 484 is Mexico, 2203 is beer): image

We need to understand why the module is replacing official data with mirror data, in both of these instances.

chrMongeau commented 7 years ago

Let's leave this on hold until we have a new run of the module. Indeed, this seems to be related to a fixed bug.

I just run the module locally and the 2010 flow for Australia is official (reporter/partner/item are FAO codes; value/quantity need to be divided by 1,000):

> tldata %>% filter(fcl == 51, reporter == 10, partner == 138)
# A tibble: 2 x 23
   year reporter partner  flow    hs6     hs qunit        value    weight       qty
  <int>    <int>   <int> <int>  <int>  <chr> <int>        <dbl>     <dbl>     <dbl>
1  2010       10     138     1 220300 220300     7 100254551.77 114816433 2289693.9
2  2010       10     138     1 220300 220300     7     18073.61        NA     285.7

so it means that the new data should be official, instead of mirrored.

chrMongeau commented 7 years ago

About the South Africa and Namibia, the problem is that the module was not excluding "So. African Customs Union" from unsdpartnerblocks: it was aggregating Botswana, Lesotho, Namibia, South Africa, and Swaziland into this area (M49 code 711) that has FAO area code 252 (other countries).

Fixed in 709bf68.