search

run-llama / llama_index

LlamaIndex is a data framework for your LLM applications
https://docs.llamaindex.ai
MIT License
33.38k stars 4.67k forks source link

[Question]: I am using Question Generator feature from llama-index , i want to know which LLM is best for this ? #13121

Open nikhitaKanoj opened 2 months ago

nikhitaKanoj commented 2 months ago

Question Validation

Question

I am using Question generator from llama index , here they use Open AI LLM , I want to use open source LLM , which is preferred? Interms of getting high accurate questions from the document uploaded : https://docs.llamaindex.ai/en/stable/examples/evaluation/QuestionGeneration/

dosubot[bot] commented 2 months ago

For generating high-accuracy questions from documents using an open-source Large Language Model (LLM), there isn't a one-size-fits-all recommendation. However, based on community discussions and issues within the LlamaIndex repository, models like GPT-J, OPT, and FLAN-T5 have been explored, with FLAN-T5 (in its xl and xxl versions) showing promising results despite limitations on input length. It's important to note that the effectiveness of these models can vary based on the specific requirements of your application, such as the need for parsing output, generating questions with guidance, or handling sub-questions.

The choice of an open-source LLM should consider the functionalities you're leveraging in LlamaIndex's Question Generator, including the ability to generate, parse, and refine questions. As the landscape of open-source LLMs is rapidly evolving, keeping an eye on the latest developments and community discussions within repositories like LlamaIndex can provide insights into new and potentially more capable models as they emerge.

If you have specific features or requirements in mind for the LLM in your question generation application, sharing those might help narrow down the options further.

#### Details **Sources** - [What are your takes on open-source model capability to implement numerous pipeline logic that you have came out with?](https://github.com/run-llama/llama_index/discussions/8449) - [llama-index tools just for open LLMs?](https://github.com/run-llama/llama_index/discussions/9356) - [Best llm to use gpt_index](https://github.com/run-llama/llama_index/issues/441) - [llama-index-core/llama_index/core/question_gen/__init__.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/question_gen/__init__.py) - [llama-index-legacy/llama_index/legacy/question_gen/__init__.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/question_gen/__init__.py) - [llama-index-finetuning/llama_index/finetuning/cross_encoders/dataset_gen.py](https://github.com/run-llama/llama_index/blob/main/llama-index-finetuning/llama_index/finetuning/cross_encoders/dataset_gen.py) - [llama-index-legacy/llama_index/legacy/finetuning/cross_encoders/dataset_gen.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/finetuning/cross_encoders/dataset_gen.py) - [docs/docs/community/integrations.md](https://github.com/run-llama/llama_index/blob/main/docs/docs/community/integrations.md) - [llama-index-integrations/llms/llama-index-llms-langchain/llama_index/llms/langchain/utils.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/llms/llama-index-llms-langchain/llama_index/llms/langchain/utils.py) - [llama-index-legacy/tests/llms/BUILD](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/tests/llms/BUILD) - [llama-index-legacy/llama_index/legacy/llms/__init__.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/llms/__init__.py) - [docs/docs/understanding/using_llms/using_llms.md](https://github.com/run-llama/llama_index/blob/main/docs/docs/understanding/using_llms/using_llms.md) - [docs/docs/module_guides/evaluating/index.md](https://github.com/run-llama/llama_index/blob/main/docs/docs/module_guides/evaluating/index.md) - [docs/docs/understanding/evaluating/evaluating.md](https://github.com/run-llama/llama_index/blob/main/docs/docs/understanding/evaluating/evaluating.md) - [llama-datasets/eval_llm_survey_paper/card.json](https://github.com/run-llama/llama_index/blob/main/llama-datasets/eval_llm_survey_paper/card.json) - [llama-index-integrations/llms/llama-index-llms-bedrock/llama_index/llms/bedrock/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/llms/llama-index-llms-bedrock/llama_index/llms/bedrock/base.py) - [llama-index-integrations/llms/llama-index-llms-openllm/pyproject.toml](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/llms/llama-index-llms-openllm/pyproject.toml) - [llama-index-integrations/llms/llama-index-llms-vllm/pyproject.toml](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/llms/llama-index-llms-vllm/pyproject.toml) - [llama-index-integrations/llms/llama-index-llms-litellm/pyproject.toml](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/llms/llama-index-llms-litellm/pyproject.toml) - [llama-index-integrations/llms/llama-index-llms-llama-api/pyproject.toml](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/llms/llama-index-llms-llama-api/pyproject.toml) - [llama-index-integrations/llms/llama-index-llms-openai-like/tests/test_llms_openai_like.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/llms/llama-index-llms-openai-like/tests/test_llms_openai_like.py)
**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

nikhitaKanoj commented 2 months ago

@dosu, My application is : When the Document is uploaded, all the possible questions has to be listed from that document. I used Nous-Hemers-Llama2 13B , the performance is very poor , whereas ofcource OpenAI LLM was veyr much accurate , I want to achieve atleast nearly accurate questions using Open source LLM's .

dosubot[bot] commented 2 months ago

For enhancing the accuracy of question generation from documents using open-source LLMs, considering your experience with Nous-Hemers-Llama2 13B, you might want to explore other models like GPT-J, OPT, or FLAN-T5. These models have shown promise in various applications, though direct comparisons with OpenAI's models in terms of question generation accuracy might be challenging due to the proprietary nature of OpenAI's technology.

To potentially achieve near OpenAI-level accuracy with open-source models, you could experiment with adjusting the configurations specific to your application's needs. This includes optimizing the input data processing, tweaking the model parameters, and refining the post-generation question filtering process. Leveraging the latest versions of these models and staying updated with the community's findings can also provide incremental improvements.

The LlamaIndex's Question Generator's design, with features like customizable prompt templates and output parsing, supports the integration of various LLMs. By experimenting with different models and configurations, you can identify the best fit for your application's requirements.

Remember, the effectiveness of an LLM in question generation can significantly depend on how well it's tuned for the specific type of documents and questions you're targeting. Continuous testing and iteration will be key to improving question generation accuracy with open-source LLMs.

#### Details **Sources** - [llama-index tools just for open LLMs?](https://github.com/run-llama/llama_index/discussions/9356) - [[Question]: How to use Azure Open AI Service with LLama Index Sub-Question?](https://github.com/run-llama/llama_index/issues/10571) - [run llama_index with locally llm that have compatible OpenAI's API](https://github.com/run-llama/llama_index/issues/1772) - [Best llm to use gpt_index](https://github.com/run-llama/llama_index/issues/441) - [llama-index-core/tests/question_gen/test_llm_generators.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/tests/question_gen/test_llm_generators.py) - [docs/docs/optimizing/production_rag.md](https://github.com/run-llama/llama_index/blob/main/docs/docs/optimizing/production_rag.md) - [llama-index-core/llama_index/core/question_gen/llm_generators.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/question_gen/llm_generators.py) - [llama-index-legacy/llama_index/legacy/question_gen/__init__.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/question_gen/__init__.py) - [llama-index-integrations/llms/llama-index-llms-openllm/llama_index/llms/openllm/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/llms/llama-index-llms-openllm/llama_index/llms/openllm/base.py)
**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.