Benchmarking guide doesn't work

[raaupr]

Hi,

Thank you for this work! It's great!

I'm trying to reproduce the benchmark results and tried to follow the guide here, but I suspect that it is not up to date? I received the following error when sending the requests:

E:Proxy:[pro:loo:110]:Request (127.0.0.1:61243): server status {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/trec_car/_search] contains unrecognized parameter: [nboost]"}],"type":"illegal_argument_exception","reason":"request [/trec_car/_search] contains unrecognized parameter: [nboost]"},"status":400}

What is the right way to do the benchmarking with the current codebase?

Thanks!

[colethienes]

Thanks! You're correct that the benchmarking hasn't been updated in a while. I will push some updates soon.

[colethienes]

Pushed version 0.2.2 with fixed benchmarking and updated docs. Let me know if you run into any issues.

[raaupr]

Thank you!!

Unfortunately, I received another error code now:

werkzeug.exceptions.MethodNotAllowed: 405 Method Not Allowed: The method is not allowed for the requested URL.

I assume that it's because the request should be get instead of post? I tried changing it to get and it seems to work, but I do not get the same page as the example when I go to http://localhost:8000/nboost. There is no mrr results, as you can see in the picture attached. Wondering if there's any config that I miss? I started nboost with option number 1 nboost --uhost <Elasticsearch host> --uport 9200

[raaupr]

Sorry!! Ignore the message above, I didn't realize I was running the version of nboost that I installed from the master branch of this repo instead of the released version 0.2.2. I will try with the released version and report back!

[raaupr]

Ok, I've tried it again. I've come across some problems:

First error:

nboost-index: error: unrecognized arguments: --name trec_car

For which I think can be fixed by running the command below instead:

nboost-index --file paragraphs.tsv --index_name trec_car --host <Elasticsearch host>

Second error happened when sending the queries:

E:Proxy:[pro:loo:121]:Request (127.0.0.1:56176): list index out of range
Traceback (most recent call last):
  File "/Users/audi/miniconda3/envs/nboost/lib/python3.7/site-packages/nboost/proxy.py", line 84, in loop
    hooks.on_rerank_response(session, self.model)
  File "/Users/audi/miniconda3/envs/nboost/lib/python3.7/site-packages/nboost/hooks.py", line 181, in on_rerank_response
    reranked_choices = [session.choices[rank] for rank in ranks]
  File "/Users/audi/miniconda3/envs/nboost/lib/python3.7/site-packages/nboost/hooks.py", line 181, in <listcomp>
    reranked_choices = [session.choices[rank] for rank in ranks]
IndexError: list index out of range

[pertschuk]

Fixed these I think.