Output is always '#' repeated

[carlos-havier]

Hi,

Cool project!

Was trying it with some books from the Gutemberg Project, for example "The Diary of a Turk by Çerkesseyhizade Halil Halit.epub".

The retrieval works well, I can see that the paragraphs with relationed info are picked up well.

Yet the output of the model is always # repeated a number of times.

Steps to reproduce:

• On Ubuntu 20.04, CUDA 12.1
• Install llm-search as indicared (in anaconda env)
• pip install pandoc
• mkdir documents
• mkdir models
• download book: cd documents; wget https://www.gutenberg.org/ebooks/50048.epub.noimages; cd ..
• download model: cd models; sudo apt-get install git-lfs; git lfs clone https://huggingface.co/TheBloke/airoboros-l2-13B-gpt4-1.4.1-GGUF; cd ..
• mv airoboros-l2-13B-gpt4-1.4.1-GGUF models
• copy sample_templates/generic/config_template.yaml config.yaml
• change config.yaml to point to models/airoboros-l2-13B-gpt4-1.4.1-GGUF/airoboros-l2-13b-gpt4-1.4.1.Q4_K_M.gguf
• change confir.yaml to point to models/intfloat/e5-large-v2
• download e5-large-v2 with the code:

from sentence_transformers import SentenceTransformer modelPath = "models/intfloat/e5-large-v2" model = SentenceTransformer('intfloat/e5-large-v2') model.save(modelPath)

Index, run web interface, and ask:

"What does Halvati mean?"

(or any other question answered in the book)

Answer:

"-###############################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################"

[snexus]

Hi,

I will try to reproduce on my end. Suspect it is a model specific issue (the model specified in the config is just an example). Could you please try some other model compatible with llama-cpp and/or change the prompt template in the config.yaml according to https://huggingface.co/TheBloke/airoboros-l2-7b-gpt4-1.4.1-GGUF#prompt-template-airoboros

[snexus]

Tried to repeat the steps - couldn't reproduce the problem. Using this model produces the following output:

Halvati means "those who worship in seclusion." The name arises from one of the strict rules of the order, that its rites must not be displayed to the outside public, doubtless a measure for the prevention of hypocrisy.

Attaching the config file I used to test it: the_diary_of_turk.tar.gz

Please configure the paths for your system, as indicated in the screenshot:

download e5-large-v2 with the code: from sentence_transformers import SentenceTransformer modelPath = "models/intfloat/e5-large-v2" model = SentenceTransformer('intfloat/e5-large-v2') model.save(modelPath)

You don't need to download the embedding model manually, the package will download and cache it for you, just specify the model name as "intfloat/e5-large-v2" in the config.

Please let me know if the problem is resolved.

[carlos-havier]

Thank you for your reply! I was out for some days but finally could come back to try your suggestions.

Indeed, I created a second Anaconda environment from scratch, downloaded again the models locally, and run the query.

In particular, I downloaded the quantized Llama model from https://huggingface.co/TheBloke/airoboros-l2-13B-gpt4-1.4.1-GGUF/tree/main. For intfloat/e5-large-v2, I did as you suggested, and let the code download the model.

I'm attaching my config.yaml file as well as the log results of the query "What does Halvati mean?". This time, the result is slightly different, it starts by writing "The" and the it follows with a long chain of "#", as you can see in the log, with some other non-printable chars in between.

At this point, I'm running out of ideas to try. If you have some, please let me know. question_log.txt conf_yaml.txt Thank you again for looking into this.

[carlos-havier]

Apologies, I forgot to mention that I also tried using your config file (updating the paths), as well as using other models from https://huggingface.co/TheBloke/airoboros-l2-13B-gpt4-1.4.1-GGUF/ (that I download using git lfs clone), but the result is always one, two or three letters, followed by unprintable chars and '#'.

I also checked the SHA256 of airoboros-l2-13b-gpt4-1.4.1.Q4_K_M.gguf as the same appearing in https://huggingface.co/TheBloke/airoboros-l2-13B-gpt4-1.4.1-GGUF/blob/main/airoboros-l2-13b-gpt4-1.4.1.Q4_K_M.gguf.

Thanks!

[carlos-havier]

Hi again! I did a test in a different computer, this time with CUDA 11.3 (instead of 12.1), and it is working without problems. I think you can close this issue. Thanks again!

[snexus]

Thanks for the test