search

abhinand5 / tamil-llama

A New Tamil Large Language Model (LLM) Based on Llama 2
GNU General Public License v3.0
255 stars 34 forks source link

Not able to get the model to give proper output #1

Closed Cyberblackstone closed 10 months ago

Cyberblackstone commented 10 months ago

Hi, I am trying to get the model to give me output on my Mac m2. I am using a quantized model specifically tamil-llama-7b-base-v0.1-q4_k_m.gguf, but I am not able to get satisfactory results as mentioned in the paper. I feel I am prompting the model incorrectly. I am using llama-cpp-python to use the model. I am a complete beginner here, so can you give a series of beginner friendly tips or instructions to get the model up and working? Thanks in advance

abhinand5 commented 10 months ago

Let's go through a few key points that might help:

  1. Foundation Model Suitability: It appears you're using a foundation model, which, as per the model card, isn't ideally suited for instruction tasks. This mismatch might be one reason why you're not seeing the expected results. Here's a visual reference from the model card for clarity: image.
    Link to Instruct 7B model -> abhinand/tamil-llama-7b-instruct-v0.1-gguf

  2. Impact of Quantization: In terms of quantization, it's important to note that reducing precision below 8 bits typically leads to a performance degradation. This could be another factor affecting the model's output quality in your case.

  3. Alternative Tools to llama.cpp: If you're looking for more beginner-friendly tools, consider exploring LMStudio or the text-generation-webui. These platforms might offer a more streamlined experience for running and interacting with language models.

  4. Using LMStudio for Beginners: Since you're a beginner, a user-friendly option could be LMStudio. You can find a sample model preset in the repository (./config/lm_studio/model_config.json). Do remember to adjust the hardware parameters in LMStudio to match your system's specifications.

I'm also planning to create a guided Colab Notebook for inferencing with the model which might help!

Cyberblackstone commented 10 months ago

HI,Thank you so much for the info! Can we further fine tune this model, and what is the process to do that?

abhinand5 commented 10 months ago

Of course you can further fine-tune the base model on a custom dataset. Please refer to the finetuning script here https://github.com/abhinand5/tamil-llama/blob/main/scripts/train/finetune/finetune.py

Cyberblackstone commented 10 months ago

Thank you so much!