search

dice-group / gerbil

GERBIL - General Entity annotatoR Benchmark
GNU Affero General Public License v3.0
224 stars 58 forks source link

Illinois Wikifier and other missing annotators #204

Closed sagnik closed 7 years ago

sagnik commented 7 years ago

I am assuming the most complete discussion of the system is in this paper published in the Semantic Web Journal, 2016? Table 3 in that paper says that 20 annotators are available n the latest version of Gerbil, but I only see approximately 10-12, notably missing Illinois Wikifier. Here's a screen shot of my configuration.

sc

Is there something wrong with my config? Also, when I generate the table at the experiment overview page, For Fred, KEA and TagMe I see error(-108). Is this expected?

RicardoUsbeck commented 7 years ago

Hi,

thanks for using GERBIL.

1) You selected the A2KB experiment type in your screenshot. Different experiment types have different annotators. Some annotators are used in more than one experiment type. So the total number stems from this configuration. 2) Illinois Wikifier API had yielded some issues in the past. We are working on it. See also #172 3) This table collect data from the latest runs for this particular type of experiment. That is, if someone run the last experiment with say KEA on a day where KEA was off or the server down, than the last database entry contains -108. So if you rerun a new experiment with KEA over all datasets in your chosen experiment type, the next time you click overview, the new stats will be present.