ceumicrodata / respect-trade-similarity

Trade Similarity Index for RESPECT project
MIT License
2 stars 2 forks source link

Fix non-numeric CN codes #6

Closed korenmiklos closed 5 years ago

korenmiklos commented 5 years ago

The following 342 codes are non-numeric in 2017 imports. (Check for other years.) Drop TOTAL and develop a strategy to deal with the rest. Check what the codes mean. If any country reports a product with masked digits (this usually happens for confidentiality reasons), we may have to aggregate that product up to the next meaningful level (4-digit or 2-digit) for all countries.

01MMM0
02MMM0
02SSS0
03MMM0
03SSS9
04MMM0
04SSS0
05MMM0
05SSS9
06MMM0
07MMM0
08MMM0
09MMM0
09SSS0
09SSS9
10MMM0
1108S5
11MMM0
11SSS0
11SSS5
12MMM0
12SSS2
12SSS9
13MMM0
13SSS2
13SSS9
14MMM0
15MMM0
15SSS0
15SSS4
15SSS9
16MMM0
1702S0
17MMM0
17SSS0
17SSS9
18MMM0
18SSS9
19MMM0
19SSS9
20MMM0
20SSS0
20SSS9
21MMM0
21SSS0
21SSS9
22MMM0
22SSS5
22SSS9
23MMM0
23SSS0
23SSS9
24BBB0
24FFF0
24MMM0
24SSS1
24SSS9
2511S2
2523S6
25MMM0
25SSS2
25SSS6
25SSS9
2615S2
26MMM0
26SSS2
26SSS9
2701S3
27BBB0
27FFF0
27MMM0
27SSS3
27SSS9
2804S5
2809S5
2825S5
2827S5
2832S5
2833S5
2834S5
2836S5
2837S5
2841S5
28MMM0
28SSS5
28SSS9
2903S5
2904S5
2905S5
2907S5
2912S5
2914S5
2915S5
2917S5
2918S5
2921S5
2922S5
2923S5
2933S5
2936S5
2939S5
29MMM0
29SSS5
29SSS9
3004S5
30MMM0
30SSS5
30SSS9
3102S5
3104S5
31MMM0
31SSS2
31SSS5
31SSS9
32MMM0
32SSS5
32SSS9
33MMM0
33SSS9
34MMM0
34SSS9
35MMM0
35SSS0
35SSS5
35SSS9
36MMM0
36SSS5
36SSS8
36SSS9
37MMM0
37SSS9
3808S5
3824S5
38MMM0
38SSS5
38SSS9
3903S5
3905S5
3907S5
3912S5
3920S5
39MMM0
39SSS5
39SSS8
39SSS9
4002S2
40MMM0
40SSS2
40SSS9
41MMM0
41SSS2
42MMM0
42SSS9
43MMM0
43SSS8
44MMM0
44SSS9
45MMM0
46MMM0
46SSS9
47MMM0
47SSS9
48MMM0
48SSS6
48SSS9
49MMM0
49SSS9
50MMM0
51MMM0
51SSS9
52MMM0
52SSS9
53MMM0
53SSS2
53SSS6
53SSS9
5402S6
54MMM0
54SSS6
54SSS9
5501S2
5503S2
55MMM0
55SSS2
55SSS9
56MMM0
56SSS6
56SSS9
57MMM0
57SSS9
58MMM0
58SSS9
59MMM0
59SSS9
60MMM0
60SSS9
61MMM0
61PPP0
61SSS9
62MMM0
62PPP0
62SSS8
62SSS9
63III0
63MMM0
63SSS9
64MMM0
64SSS9
65MMM0
65SSS9
66MMM0
66SSS9
67MMM0
67SSS9
6806S6
68III0
68MMM0
68SSS6
68SSS9
69III0
69MMM0
69SSS6
69SSS8
69SSS9
7002S6
70III0
70MMM0
70SSS6
70SSS9
71MMM0
71SSS2
71SSS6
71SSS9
72III0
72MMM0
72SSS6
72SSS9
7326S6
73CCC0
73III0
73MMM0
73SSS6
73SSS9
74MMM0
74SSS6
74SSS9
75MMM0
75SSS6
76III0
76MMM0
76SSS6
76SSS9
78MMM0
79MMM0
79SSS6
79SSS9
80MMM0
8101S6
8103S6
81MMM0
81SSS6
81SSS9
82III0
82MMM0
82PPP0
82SSS6
82SSS9
83MMM0
83SSS9
8424S7
8448S7
8467S7
84CCC0
84III0
84MMM0
84SSS7
84SSS9
85CCC0
85III0
85MMM0
85PPP0
85SSS7
85SSS9
86III0
86MMM0
86SSS9
8710S8
87III0
87MMM0
87SSS7
87SSS8
87SSS9
87VVV0
8802S7
88MMM0
88SSS7
88SSS8
88SSS9
88WWW0
89MMM0
89SSS7
89SSS9
9013S8
9022S7
90III0
90MMM0
90SSS7
90SSS8
90SSS9
91MMM0
91SSS8
92MMM0
92SSS8
92SSS9
9301S8
9302S8
9305S8
9306S8
93MMM0
93SSS8
93SSS9
94III0
94MMM0
94SSS9
95MMM0
95SSS8
95SSS9
96MMM0
96SSS8
96SSS9
97MMM0
97SSS9
99AAA0
99BBB0
99CCC0
99FFF0
99III0
99PPP0
99RRR1
99SSS9
99YYY0
TOTAL
korenmiklos commented 5 years ago

@gaborberei I have recomputed without the confidential product codes trade_data_filtered, see . The bead is in /srv/dropbox_encrypted/bead-box. Please update your input bead, but note that the filenames have changed to full20XX52.csv. The column names are the same. Then rerun the index calculator. The change in index values should be minimal.

korenmiklos commented 5 years ago

I also dropped rows with PRODUCT_CN=="TOTAL".