search

hosseinfani / ReQue

A Benchmark Workflow and Dataset Collection for Query Refinement
https://hosseinfani.github.io/ReQue/
Other
25 stars 1 forks source link

Comparison of Sep 3 ReQue Results to Results From Demonstration Video - Please Comment on the Differences #4

Closed kpoots closed 4 years ago

kpoots commented 4 years ago

The attached image is a small snapshot of query expansion results for the dataset robust04, my results vs Demonstration Video.

The results are similar but not the same. I have circled in red some of the more interesting differences.

I suspect that this shows how ReQue has evolved over time, but I'm not sure.

When there is time, I ask for comment from the ReQue development team.

Thank you.

Kent image

hosseinfani commented 4 years ago

Thanks to raise this up. Our expanders rely on external libraries, e.g., in wordnet expander we rely on nltk.corpus v3.4 in here. Updates to those corpora may change the final result if you do not use the same version we used in our codeline.

Even if you use the exact versions of libraries in our codeline, you still might see some inconsistencies for few expanders like thesaurus. The reason is that we are reaching www.thesaurus.com to find the synonyms. Some connections in a run may be failed due to high load to the website or else. So the result of this expander is not deterministic due to connection issues in the internet, etc.

Hope it answers your issue.

kpoots commented 4 years ago 
                                                                                  Yes, that's good, thank you!                                                                                                                                                                                                                                                                                                                                         Sent from my BlackBerry 10 smartphone on the Rogers network.                                                                                                                                                                                                                From: Hossein FaniSent: Friday, September 4, 2020 3:58 PMTo: hosseinfani/ReQueReply To: hosseinfani/ReQueCc: kent poots; AuthorSubject: Re: [hosseinfani/ReQue] Comparison of Sep 3 ReQue Results to Results From Demonstration Video - Please Comment on the Differences (#4)

Thanks to raise this up. Our expanders rely on external libraries, e.g., in wordnet expander we rely on nltk.corpus v3.4 in here. Updates to those corpora may change the final result if you do not use the same version we used in our codeline. Even if you use the exact versions of libraries in our codeline, you still might see some inconsistencies for few expanders like thesaurus. The reason is that we are reaching www.thesaurus.com to find the synonyms. Some connections in a run may be failed due to high load to the website or else. So the result of this expander is not deterministic due to connection issues in the internet, etc. Hope it answers your issue.

—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe. [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/hosseinfani/ReQue/issues/4#issuecomment-687352181", "url": "https://github.com/hosseinfani/ReQue/issues/4#issuecomment-687352181", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]