Why are the eci computed with ecomplexity different from their given values?

[chengjun]

Thank you for contributing this brilliant python package. I am trying to compute the eci (eci_ecomplexity_cal) using the country_sitcproduct2digit_year.csv data. In the country_sitcproduct2digit_year.csv data, there are given eci values (eci_hidalgo_rep) . I found that they do not line up exactly.

Further I find that the given eci values (eci_hidalgo_rep) has a better correlation with the GDP per capita compared with the eci computed using this python package (eci_ecomplexity_cal).

[shreyasgm]

Thanks for bringing this to our attention. We'll look into this.

Here is code to reproduce the issue:

import pandas as pd
import numpy as np
import seaborn as sns
from ecomplexity import ecomplexity

# Import trade data from CID Atlas
data_url = "https://intl-atlas-downloads.s3.amazonaws.com/country_sitcproduct2digit_year.csv.zip"
data = pd.read_csv(data_url, compression="zip", low_memory=False)
data.head()

# Calculate eci's
data = data[['year','location_code','sitc_product_code','export_value','sitc_eci']]
data = data[data.year==data.year.max()]
trade_cols = {'time':'year', 'loc':'location_code', 'prod':'sitc_product_code', 'val':'export_value'}
cdata = ecomplexity(data, trade_cols)
cdata = cdata[['location_code','eci','sitc_eci']].drop_duplicates(['location_code'])

# Compare against Atlas
sns.scatterplot('sitc_eci','eci', data=cdata)

[shreyasgm]

Ah, this issue is much simpler than I expected. I didn't notice earlier, but you're using 2-digit SITC codes. The calculations for ECI in the Atlas are based on 4-digit SITC codes. This explains most of the discrepancy that you're seeing. Any remaining differences are due to the fact that on the Atlas, we restrict the set of countries to ones with reliable data for ECI calculations. Learn more about Atlas data here

Closing as not an issue.

[chengjun]

@shreyasgm Thank you for your generous and quick reply. I have just tried the 4-digit SITC dataset using your python script as follows.

import pandas as pd
import numpy as np
import seaborn as sns
from ecomplexity import ecomplexity

# Import trade data from CID Atlas
data = pd.read_csv('../data/country_partner_sitcproduct4digit_year_2017.zip', compression="zip", low_memory=False)
data.head()

# Calculate eci's
data = data[['year','location_code','sitc_product_code','export_value','sitc_eci']]
data = data[data.year==data.year.max()]
trade_cols = {'time':'year', 'loc':'location_code', 'prod':'sitc_product_code', 'val':'export_value'}
cdata = ecomplexity(data, trade_cols)
cdata = cdata[['location_code','eci','sitc_eci']].drop_duplicates(['location_code'])

# Compare against Atlas
# To download gdp data from wordbank 
# https://data.worldbank.org/indicator/ny.gdp.pcap.cd?most_recent_year_desc=true
df_gdp = pd.read_csv('./GDP/API_NY.GDP.PCAP.CD_DS2_en_csv_v2_41089.csv', sep = '\t')
year =2017
df_g = df_gdp[['Country Code', str(year)]]
gdp_dict =dict(zip(df_g['Country Code'],df_g[str(year)]))
cdata['gdp'] = [gdp_dict[name] if name in gdp_dict else 0.0 for name in cdata['location_code']]

# Compare against Atlas
df = cdata[cdata['gdp'] >0]
sns.scatterplot('eci','gdp', data=df, label = 'eci computed with ecomplexity')
sns.scatterplot('sitc_eci','gdp', data=df, label = 'sitc_eci given by Hidalgo')
plt.yscale('log')
plt.legend()
plt.xlabel('ECI', fontsize = 15)
plt.ylabel('GDP', fontsize = 15)
plt.title('2017');

I think this problem has been largely solved following your suggestion. Thank you for your comments.

[shreyasgm]

I did mention earlier that there would be some remaining differences. Atlas data uses a set of processes that clean the trade data based on bilateral reporting, and restrict the set of countries used for ECI calculations. More here. If you have specific questions about the Atlas data itself, I would suggest that you request help from the Atlas team here.

[chengjun]

@shreyasgm Thank you for your kindly reply. I have benefited a lot from your insightful and comprehensive answers which have well addressed my question. The idea of ECI is very inspiring and it could be applied to many other areas. Thanks again for your maintaining of this wonderful Python package. Best regards. C.J.