search

dice-group / gerbil

GERBIL - General Entity annotatoR Benchmark
GNU Affero General Public License v3.0
219 stars 57 forks source link

experiment with errors #311

Open tarcisiosouza opened 5 years ago

tarcisiosouza commented 5 years ago

Hi,

I submitted a job in gerbil with 3 questions to test and after a couple of times getting errors such as "The annotator caused too many single errors", now the job seems to be running indefinitely.

You can follow it here: http://gerbil-qa.aksw.org/gerbil/experiment?id=201902210063

If it finishes, it will probably show that error I mentioned. Can someone tell me why is it still running and point out what are exactly the errors about?

I uploaded my own dataset, but even if I use one of the available datasets, as the QALD6 Test Multilingual, the same thing happens.

Thank you.

TortugaAttack commented 5 years ago

Hi, I will check that out. Just in case: Can you provide me the file? (3 questions is easier to debug than a complete QALD 6 file) Otherwise i will just check with the QALD 6 dataset.

tarcisiosouza commented 5 years ago

Thank you. I attached the file with 3 questions. eventqa-train-questions.txt

TortugaAttack commented 5 years ago

YODA does not work since 1-2 years or so. they shut down their demo which was working with gerbil and replaced it with some strange one. SINA had always problems, HAWK and QANARA/wdaqua changed url/ip OKBQA changes url also very often. (i did not check, but it seems to be the problem) ganswer2 seems to be down from the debug messages.

all new IPs are not in the online release yet.

and if i saw correctly: the max amount of errors is set to 1. @MichaelRoeder is that correct? it seems off to me.

MichaelRoeder commented 5 years ago

There is a maximum amount of questions that are allowed to fail. This is currently hard coded to 0.25. If 1of 3 questions causes an error (i.e., an exception) this threshold is already reached.

TortugaAttack commented 5 years ago

OK. So we just need to change the URLs to the correct ones. We have to think of a better way than change the URLs every half a year :/

tarcisiosouza commented 5 years ago

Ok. That means that my sample is correct, right? Did you guys update the URLs already? I was testing again with same errors. Thank you very much.

RicardoUsbeck commented 5 years ago

ping @TortugaAttack

TortugaAttack commented 5 years ago

I updated every URL/IP i could find, and pushed it, did not update the online version yet though. Will do that today then close the issue.

tarcisiosouza commented 5 years ago

Dear all,

Is it true that only QUEPY system is working for QA in gerbil after the URLs update? I uploaded my dataset and could get some results with 24 errors only by removing the dates in my queries with QUEPY system. Can you tell me what are theses errors about so that I can adapt my data accordingly if needed?

You see the results for this in the URL: http://gerbil-qa.aksw.org/gerbil/experiment?id=201904080006

Best regards!

Tarcísio Souza PhD Candidate L3S Research Center / Leibniz Uni. 15th Floor, Room: 1519 Appelstraße 9 30167 Hannover, Germany Am 6. Apr. 2019, 14:39 +0200 schrieb F. Conrads notifications@github.com:

Closed #311. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

RicardoUsbeck commented 5 years ago

Hi, indeed these numbers do not look ok. @LukasBluebaum can you have a look @TortugaAttack maybe you can upload the log information here? Do not know whether this is useful or not.

LukasBluebaum commented 5 years ago

Yes, I will take a look.