search

uclnlp / simpleNumericalFactChecker

Fact checker for simple claims about statistical properties
26 stars 5 forks source link

Obtaining the real vs. predicted EV pairs for each pattern #3

Closed dhruvghulati-zz closed 8 years ago

dhruvghulati-zz commented 8 years ago

It seems like this is where you can obtain the real vs. predicted EV pairs after training, after adding each pattern. Is this correct? My aim is to obtain for each pattern, what the final real vs. predicted EV pairs are. It seems like when I use your adjusted MAPE code, you run through testing each pattern using different params (e.g. 0.0625, 32) and then have EV predictions for each param and pattern set. You then have a bestParams parameter for all of these predictions? How can I obtain the adjustedMAPE for each pattern using the bestParams you already got to?

e.g. when I uncomment these lines, I get:

real: {u'Canada': 52218.99, u'Afghanistan': 619.59, u'Madagascar': 447.44, u'Turkmenistan': 6510.61, u'Liberia': 421.7, u'Guinea': 591.02, u'Vanuatu': 3176.21, u'Cambodia': 945.99, u'Swaziland': 3043.5, u'Laos': 1399.21, u'Seychelles': 11758.04, u'Cameroon': 1151.36, u'Burkina Faso': 634.32, u'Ecuador': 5456.43, u'Bahrain': 18334.17, u'Brunei': 41126.61, u'Saudi Arabia': 20777.67, u'Belarus': 6685.02, u'Algeria': 5404.0, u'Slovenia': 22092.26, u'Venezuela': 12766.72, u'Zambia': 1469.12, u'Montenegro': 6813.04, u'Papua New Guinea': 2184.16, u'Togo': 574.12, u'Zimbabwe': 787.94, u'Kiribati': 1743.39, u'Haiti': 770.95, u'Belize': 4576.64, u'Hong Kong': 36795.82, u'C\xf4te d\u2019Ivoire': 1243.99, u'Ukraine': 3866.99, u'Mauritania': 1106.14, u'Tonga': 4493.87, u'Tanzania': 608.85, u'Israel': 31281.47, u'Mali': 693.98, u'Philippines': 2587.88, u'Sweden': 55244.65, u'Latvia': 14008.51, u'Gabon': 11430.49, u'Guyana': 3583.96, u'Thailand': 5473.75, u'Switzerland': 79052.34, u'Bulgaria': 6986.04, u'Iraq': 6454.62, u'Honduras': 2264.09, u'Macau': 78275.15, u'Chad': 885.11, u'United Arab Emirates': 40363.16, u'United Kingdom': 38514.46, u'Malaysia': 10380.54, u'Vietnam': 1595.81, u'Saint Vincent and the Grenadines': 6515.22, u'Uganda': 547.01, u'South Korea': 23020.0, u'Cyprus': 26315.47, u'Barbados': 13076.46} predicted: {u'Canada': 52147.75, u'Afghanistan': 0, u'Madagascar': 445.5, u'Turkmenistan': 6972.75, u'Mauritania': 0, u'Guinea': 571.75, u'Vanuatu': 0, u'Cambodia': 976.25, u'Swaziland': 0, u'Laos': 0, u'Venezuela': 12726.5, u'Burkina Faso': 0, u'Ecuador': 5743.0, u'Bahrain': 0, u'Brunei': 40499.0, u'Saudi Arabia': 24979.75, u'Belarus': 0, u'Algeria': 5466.25, u'Togo': 0, u'Cameroon': 1206.0, u'Zambia': 0, u'Montenegro': 0, u'Papua New Guinea': 2236.2049999999999, u'Slovenia': 0, u'Zimbabwe': 0, u'Kiribati': 0, u'Haiti': 774.0, u'Belize': 0, u'Hong Kong': 37375.0, u'Tanzania': 0, u'Ukraine': 3864.5, u'Liberia': 0, u'Tonga': 0, u'Iraq': 0, u'C\xf4te d\u2019Ivoire': 0, u'Israel': 34305.5, u'Philippines': 0, u'Sweden': 56356.5, u'Latvia': 0, u'Gabon': 0, u'Guyana': 0, u'Mali': 685.25, u'Switzerland': 80024.75, u'Thailand': 5630.75, u'Bulgaria': 7154.5, u'Seychelles': 0, u'Honduras': 0, u'Chad': 0, u'Macau': 81283.333333333328, u'United Arab Emirates': 0, u'United Kingdom': 38797.473333333328, u'Malaysia': 10437.5, u'Vietnam': 0, u'Saint Vincent and the Grenadines': 0, u'Uganda': 0, u'South Korea': 0, u'Cyprus': 0, u'Barbados': 0} printed to console.

Crucially, how do I link these lines of code to each pattern?

andreasvlachos commented 8 years ago

Not sure I understand, which lines are you referring to?

adjustedMAPE is only used in ordering the patterns when selecting which ones to keep. For evaluation, only MAPE makes sense, perhaps combined with the coverage, as done in section 4 in my paper.