search

starsuzi / Adaptive-RAG

Apache License 2.0
147 stars 21 forks source link

Usage for gpt-3.5-turbo-instruct #9

Closed oneonlee closed 1 month ago

oneonlee commented 2 months ago

Hello @starsuzi. Thanks for sharing the code.

In the paper, you measured the performance using gpt-3.5-turbo-instruct, but the published code doesn't seem to use gpt.

The reason for this expectation is as follows:

The valid_model_shortnames in llm_server/serve.py does not include "gpt-3.5-turbo-instruct", so I assumed that llm_server cannot be used if "gpt-3.5-turbo-instruct" is used as a generator. If my guess is correct, it should not be able to run run_retrieval_test.sh when using "gpt-3.5-turbo-instruct" as a generator.

I would like to know if it is possible to use "gpt-3.5-turbo-instruct" as a generator in your published code. If I'm missing something, please let me know.

Thanks.

Korean

안녕하세요. 논문 발표와 코드 공유에 감사드립니다. 논문에서는 gpt-3.5-turbo-instruct를 사용하여 성능을 측정한 결과가 있지만, 공개된 코드에는 gpt 사용이 불가능해보입니다. 이렇게 예상한 이유는 아래와 같습니다. [`llm_server/serve.py`의 `valid_model_shortnames`](https://github.com/starsuzi/Adaptive-RAG/blob/main/llm_server/serve.py#L35)에 "gpt-3.5-turbo-instruct"가 포함되어 있지 않으므로, "gpt-3.5-turbo-instruct"를 generator로 사용하는 경우에는 llm_server를 사용할 수 없다고 추측하였습니다. 만약 제 추측이 맞다면, "gpt-3.5-turbo-instruct"를 generator로 사용하는 경우에는 run_retrieval_test.sh를 실행할 수 없을 것 같습니다. 공개된 코드에서 "gpt-3.5-turbo-instruct"를 generator로 사용가능한지 문의드립니다. 혹시 제가 놓친게 있다면 알려주시면 감사하겠습니다.

starsuzi commented 1 month ago

Hello @oneonlee, thanks for your interest in our work!

You can refer to https://github.com/starsuzi/Adaptive-RAG/blob/main/commaqa/models/gpt3generator.py for the gpt model, where each config file for each setting for various models (including the GPT model) is stored in https://github.com/starsuzi/Adaptive-RAG/tree/main/base_configs.