dice-group / GerbilQA-Benchmarking-Template

A template for QA systems to benchmark with GERBIL
GNU Affero General Public License v3.0
0 stars 2 forks source link

Current response doesn't comply with Gerbil's desired response format. #4

Closed rrichajalota closed 6 years ago

rrichajalota commented 6 years ago

When tested against Gerbil : http://gerbil-qa.aksw.org/gerbil/experiment?id=201801300000

Current response:

{
    "questions": [{
        "question": [{
            "string": "What is the capital of Germany?",
            "language": "en"
        }],
        "answertype": "resource",
        "query": {
            "sparql": "SELECT ?s {?s ?p <http://example.com/prop/staticClass>} LIMIT 3"
        },
        "answers": [{
            "head": {
                "vars": ["resource"]
            },
            "results": {
                "bindings": [{
                    "resource": {
                        "type": "uri",
                        "value": "http://example.com/res/static-1"
                    }
                }, {
                    "resource": {
                        "type": "uri",
                        "value": "http://example.com/res/static-3"
                    }
                }, {
                    "resource": {
                        "type": "uri",
                        "value": "http://example.com/res/static-2"
                    }
                }]
            }
        }],
        "id": "1"
    }]
}

Expected response:

{
    "questions": [{
        "id": "1",
        "answertype": "resource",
        "question": [{
            "language": "en",
            "string": "What is the capital of Germany?"
        }],
        "query": {
            "sparql": "SELECT ?s {?s ?p <http://example.com/prop/staticClass>} LIMIT 3"
        },
        "answers": [{
            "head": {
                "vars": [
                    "uri"
                ]
            },
            "results": {
                "bindings": [{
                        "uri": {
                            "type": "uri",
                            "value": "http://example.com/res/static-1"
                        }
                    },
                    {
                        "uri": {
                            "type": "uri",
                            "value": "http://example.com/res/static-2"
                        }
                    },
                    {
                        "uri": {
                            "type": "uri",
                            "value": "http://example.com/res/static-3"
                        }
                    }
                ]
            }
        }]
    }]
}
RicardoUsbeck commented 6 years ago

Can you please try again? @rrichajalota ?

rrichajalota commented 6 years ago

@RicardoUsbeck I tried and it still gives the same error. http://gerbil-qa.aksw.org/gerbil/experiment?id=201802140002

TortugaAttack commented 6 years ago

where exactly is the difference? (Except from the different name of the variable?) Cannot see it :/

TortugaAttack commented 6 years ago

Okay, I diffed them and it seems the order is different (which should not be a problem as it is JSON) and the varname is different, which if this is the problem we have to change in either GerbilQA or qa.commons I will check for logs in Gerbil and see whats exactly wrong. ;)

TortugaAttack commented 6 years ago

Small Datasets works for me just fine. screenshot-2018-2-15 http localhost

Maybe we do not have the most recent version of gerbil QA running? If i remember correctly there was a bug fix recently with QALD WS which caused this. So basically we just need to update Gerbil QA to the last fix ;)

RicardoUsbeck commented 6 years ago

@MichaelRoeder :D ?

MichaelRoeder commented 6 years ago

Services has been updated and restarted. Please try again.

TortugaAttack commented 6 years ago

done. (tested it with localhost cause i did not want to forward a port on my flatmates network :D) screenshot-2018-2-21 http localhost