llm-rag-langchain answer wrongly when using llama-3-8b-instruct

[openvino-book]

Describe the bug llm-rag-langchain answer wrongly when using llama-3-8b-instruct + bge-small-en-v1.5 model + bge-reranker-large The llama-3-8b-instruct is quantized to INT4

All models are loaded into CPU bge-small-en-v1.5 model looks being loaded successfully.

bge-reranker-large model looks being loaded successfully.

llama-3-8b-instruct INT4 model looks being loaded successfully.

Bug: The generated answer should be correct, however, it's totally wrong!

Installation instructions (Please mark the checkbox) [ √ ] I followed the installation guide at https://github.com/openvinotoolkit/openvino_notebooks#-installation-guide to install the notebooks.

Environment information Please run python check_install.py in the _openvinonotebooks directory. If the output is NOT OK for any of the checks, please follow the instructions to fix that. If that does not work, or if you still encounter the issue, please paste the output of check_install.py here.

Additional context Add any other context about the problem here.

[OpenVINO-dev-contest]

hi @openvino-book could you click on the Hide searching result in prompt and then try it again ?

It works good on my side.

[OpenVINO-dev-contest]

Here my langchain and Transformers lib version, for your reference

langchain==0.2.6
langchain-community==0.2.6
langchain-core==0.2.11
langchain-text-splitters==0.2.2
langchainhub==0.1.20
llama-index-llms-langchain==0.1.4

sentence-transformers==3.0.1
transformers==4.40.2
transformers-stream-generator==0.0.5

[openvino-book]

Here my langchain and Transformers lib version, for your reference

langchain==0.2.6
langchain-community==0.2.6
langchain-core==0.2.11
langchain-text-splitters==0.2.2
langchainhub==0.1.20
llama-index-llms-langchain==0.1.4

sentence-transformers==3.0.1
transformers==4.40.2
transformers-stream-generator==0.0.5

That's helpful! I guess I use the Ali source, which keeps the version of transformer as 4.32.1

I'll try it again, let you know the result

[openvino-book]

Finally, the problem has been solved, thank you @OpenVINO-dev-contest very much!

[openvino-book]

The key Python packages should be

transformers==4.41.2 optimum-intel==1.18.0 openvino-tokenizers==2024.2.0.0 openvino==2024.2.0 nncf==2.11.0 langchain==0.2.6 langchain-community==0.2.6 langchain-core==0.2.11 langchain-text-splitters==0.2.2 langchainhub==0.1.20

The Llama3 weights should choose the certifed one from LLM Research