[TIARA] Get unexpected results

[cingtiye]

Hi This article is very creative. When I execute the following code,

python algorithm/grailqa_generation.py --prompt lf_schema

I get the following results:

>> root@train-grailqa-0:/data1/Projects/KC/papers/TIARA/src# python utils/statistics/grailqa_evaluate.py ../dataset/GrailQA/grailqa_v1.0_dev.json ../logs/grailqa_dev_2023_02_01_14_59_42_log.json

{'em': 0.6590270589974864, 'f1': 0.7318337303152712, 'em_iid': 0.770872567482737, 'f1_iid': 0.8097763582287697, 'em_comp': 0.5977542932628798, 'f1_comp': 0.6808620585344852, 'em_zero': 0.6356673960612691, 'f1_zero': 0.7189804767074774}

Could you please upload your dev result file? I can't sure whether my freebase is set correctly.

Thanks.

[yhshu]

Thanks for your interests.

Could you provide me with an e-mail? So I can send you a log. We use the Freebase as the README says.

[cingtiye]

Thanks for your interests.

Could you provide me with an e-mail? So I can send you a log. We use the Freebase as the README says.

Thank you very much! cingtiye@163.com

[cingtiye]

Hi yhshu,

I found some logical form from your bog can't be executed by my Freebase, such as

"(AND cvg.computer_videogame (JOIN cvg.computer_videogame.release_date 2010-03-28^^http://www.w3.org/2001/XMLSchema#date))"

The above results are as follows.

PREFIX ns: <http://rdf.freebase.com/ns/>
SELECT DISTINCT ?x
WHERE {
FILTER (!isLiteral(?x) OR lang(?x) = '' OR langMatches(lang(?x), 'en'))
?x ns:cvg.computer_videogame.release_date "2010-03-28-08:00"^^<http://www.w3.org/2001/XMLSchema#date> .
?x ns:type.object.type ns:cvg.computer_videogame .
}

Could you provide me some solutions?

Best,

[yhshu]

In my log, this logical form actually have answers. I guess there are two issues to check:

1. make sure you're using the execution function in this repo
2. make sure this Freebase dump is used

[cingtiye]

In my log, this logical form actually have answers. I guess there are two issues to check:

1. make sure you're using the execution function in this repo
2. make sure this Freebase dump is used

I'm sure that I used the execution in this repo and my Freebase dump is used. But some logical form can't still executed.

[yhshu]

-08:00 should be removed, and the answers can be obtained. I would check the code some times later.

[cingtiye]

Thank you very much. When I remove '-08:00' and the the answers can be obtained.

I check the code in grailqa_generation.py, in which the input of the function self.retriever.query_var includes the string x. x may lack practical meaning.

[yhshu]

After lisp to SPARQL process in GrailQA, the var in SPARQL is x. But if you'd like to write your own SPARQL outside of these datasets, you can change x to any var you specify.

[cingtiye]

After lisp to SPARQL process in GrailQA, the var in SPARQL is x. But if you'd like to write your own SPARQL outside of these datasets, you can change x to any var you specify.

Thank you very much. I understand. The code removes '-08:00' only to get answer, however the logical form still reserves '-08:00'.

[yhshu]

Thank you for your report. In order to get the most accurate fix based on your experiments, I wonder if you could initiate a pull request? Thanks.

[cingtiye]

Thank you for your report. In order to get the most accurate fix based on your experiments, I wonder if you could initiate a pull request? Thanks.

okk.