search

percyliang / sempre

Semantic Parser with Execution
Other
832 stars 299 forks source link

Benchmarking sempre and parasempre 1.0 #59

Open ysu1989 opened 9 years ago

ysu1989 commented 9 years ago

Hi,

I'd like to benchmark sempre and parasempre with a new set of ground-truth questions to evaluate the accuracy and runtime of the systems. I'm using sempre 1.0 so that I can reproduce the results in the papers. My question is concerned with the parameters to run the systems, and it consists of two parts:

  1. If I test the systems with pre-trained models (e.g., the model trained on WebQuestions), what's the most appropriate configuration for each system in order for them to achieve their best performance (in both accuracy and runtime).
  2. If I split the new question set into a training set and a test set, what's the best configuration for each system then for both training and testing?

Thanks in advance for taking care of the issue, and feel free to let me know if there is any confusion in the question.

RobinCai1993 commented 8 years ago

Do you know where to download sempre 1.0?

ysu1989 commented 8 years ago

You can find it as a previous release: https://github.com/percyliang/sempre/releases

RobinCai1993 commented 8 years ago

Thank you! And did you work it on mac os or Ubuntu? Because I am using mac and can not get through the 'make install' step of "install database". Could you give me some suggestions?

ysu1989 commented 8 years ago

I used Ubuntu. I guess Linux is suggested. Not sure whether it has been tested on Mac.

On Thu, Oct 27, 2016 at 10:04 PM, RobinCai1993 notifications@github.com wrote:

Thank you! And did you work it on mac os or Ubuntu? Because I am using mac and can not get through the 'make install' step of "install database". Could you give me some suggestions?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/percyliang/sempre/issues/59#issuecomment-256837048, or mute the thread https://github.com/notifications/unsubscribe-auth/AEJdlAtQ342tW1l7o12U3WtTFvD8KD_3ks5q4YJYgaJpZM4E_8JH .

Yu Su

PhD Candidate, Department of Computer Science

University of California, Santa Barbara

CA, 93117. United States.

Tel: (805)886-7320

RobinCai1993 commented 8 years ago

Thank you! I've solved it by using "brew install virtuoso" and change the content of virtuoso.init file in the installed path.