search

UKPLab / coling2018-graph-neural-networks-question-answering

Accompanying code for our COLING 2018 paper "Modeling Semantics with Gated Graph Neural Networks for Knowledge Base Question Answering"
Apache License 2.0
173 stars 29 forks source link

Replication of paper results #16

Open gtesei opened 4 years ago

gtesei commented 4 years ago

Hello,

I would like to replicate the results in the paper. I used a simple bash script to generate the results. 

#!/bin/bash

models=$(find trainedmodels/DS_COLING_2018_QA_models -name *.pkl)

for mod in $models
do
    echo ">> replicating results of $mod on websp..."
    python -m questionanswering.evaluate_on_test $mod  configs/webqsp_eval_config.yaml
    echo ">> replicating results of $mod on qald..."
    python -m questionanswering.evaluate_on_test $mod  configs/qald_eval_config_2.yaml
done

I got these results, that don't look like the ones in the paper. 

STAGGModel_2018-03-14_811799,STAGGModel,Simple,-1,webqsp,full,NoEntityList,0.0,0.00968054211035818,0.0
STAGGModel_2018-03-14_811799,STAGGModel,Simple,container description not available,-1,webqsp,0,NoEntityList,0.0,0.002702702702702703,0.0,
STAGGModel_2018-03-14_811799,STAGGModel,Simple,container description not available,-1,webqsp,1,NoEntityList,0.0,0.013953488372093023,0.0,
STAGGModel_2018-03-14_811799,STAGGModel,Simple,container description not available,-1,webqsp,2,NoEntityList,0.0,0.04081632653061224,0.0,
STAGGModel_2018-03-14_811799,STAGGModel,Simple,container description not available,-1,webqsp,3,NoEntityList,0.0,0.10344827586206896,0.0,
STAGGModel_2018-03-14_811799,STAGGModel,Simple,-1,qald,full,NoEntityList,0.0,0.0,0.0
GNNModel_2018-03-15_369757,GNNModel,Simple,-1,webqsp,full,NoEntityList,0.0,0.00968054211035818,0.0
GNNModel_2018-03-15_369757,GNNModel,Simple,container description not available,-1,webqsp,0,NoEntityList,0.0,0.002702702702702703,0.0,
GNNModel_2018-03-15_369757,GNNModel,Simple,container description not available,-1,webqsp,1,NoEntityList,0.0,0.013953488372093023,0.0,
GNNModel_2018-03-15_369757,GNNModel,Simple,container description not available,-1,webqsp,2,NoEntityList,0.0,0.04081632653061224,0.0,
GNNModel_2018-03-15_369757,GNNModel,Simple,container description not available,-1,webqsp,3,NoEntityList,0.0,0.10344827586206896,0.0,
GNNModel_2018-03-15_369757,GNNModel,Simple,-1,qald,full,NoEntityList,0.0,0.0,0.0
PooledEdgesModel_2018-03-13_519272,PooledEdgesModel,Simple,-1,webqsp,full,NoEntityList,0.0,0.00968054211035818,0.0
PooledEdgesModel_2018-03-13_519272,PooledEdgesModel,Simple,container description not available,-1,webqsp,0,NoEntityList,0.0,0.002702702702702703,0.0,
PooledEdgesModel_2018-03-13_519272,PooledEdgesModel,Simple,container description not available,-1,webqsp,1,NoEntityList,0.0,0.013953488372093023,0.0,
PooledEdgesModel_2018-03-13_519272,PooledEdgesModel,Simple,container description not available,-1,webqsp,2,NoEntityList,0.0,0.04081632653061224,0.0,
PooledEdgesModel_2018-03-13_519272,PooledEdgesModel,Simple,container description not available,-1,webqsp,3,NoEntityList,0.0,0.10344827586206896,0.0,
PooledEdgesModel_2018-03-13_519272,PooledEdgesModel,Simple,-1,qald,full,NoEntityList,0.0,0.0,0.0
OneEdgeModel_2018-03-14_194679,OneEdgeModel,Simple,-1,webqsp,full,NoEntityList,0.0,0.00968054211035818,0.0
OneEdgeModel_2018-03-14_194679,OneEdgeModel,Simple,container description not available,-1,webqsp,0,NoEntityList,0.0,0.002702702702702703,0.0,
OneEdgeModel_2018-03-14_194679,OneEdgeModel,Simple,container description not available,-1,webqsp,1,NoEntityList,0.0,0.013953488372093023,0.0,
OneEdgeModel_2018-03-14_194679,OneEdgeModel,Simple,container description not available,-1,webqsp,2,NoEntityList,0.0,0.04081632653061224,0.0,
OneEdgeModel_2018-03-14_194679,OneEdgeModel,Simple,container description not available,-1,webqsp,3,NoEntityList,0.0,0.10344827586206896,0.0,
OneEdgeModel_2018-03-14_194679,OneEdgeModel,Simple,-1,qald,full,NoEntityList,0.0,0.0,0.0
GNNModel_2018-03-13_113541,GNNModel,Gated,-1,webqsp,full,NoEntityList,0.0,0.00968054211035818,0.0
GNNModel_2018-03-13_113541,GNNModel,Gated,container description not available,-1,webqsp,0,NoEntityList,0.0,0.002702702702702703,0.0,
GNNModel_2018-03-13_113541,GNNModel,Gated,container description not available,-1,webqsp,1,NoEntityList,0.0,0.013953488372093023,0.0,
GNNModel_2018-03-13_113541,GNNModel,Gated,container description not available,-1,webqsp,2,NoEntityList,0.0,0.04081632653061224,0.0,
GNNModel_2018-03-13_113541,GNNModel,Gated,container description not available,-1,webqsp,3,NoEntityList,0.0,0.10344827586206896,0.0,
GNNModel_2018-03-13_113541,GNNModel,Gated,-1,qald,full,NoEntityList,0.0,0.0,0.0

Any thought? ...

Thank you for any help.

PS I also modified questionanswering.evaluate_on_test replacing container.description with container description not available to avoid an error from fackel

freyaya123 commented 4 years ago

have you changed the backend of Wikidata? like, backend: "http://localhost:8890/sparql"

blackkingback commented 4 years ago

Hey.Can I ask a question?What gpu did you use on this test?I keep getiing running out of memory error.