CodeLlama Instruct Configuration

[schauppi]

Hello Guys,

Could you guide me in the right direction to get the configuration of the Code Llama Instruct model right?

I have this config so far:

 {
    "name": "Code Llama",
    "endpoints": [{"url": "http://127.0.0.1:8080"}],
    "description": "Programming Assistant",
    "userMessageToken": "[INST]",

    "assistantMessageToken": "[/INST]",

    "parameters": {
      "temperature": 0.9,
      "top_p": 0.95,
      "repetition_penalty": 1.2,
      "top_k": 50,
      "truncate": 1000,
      "max_new_tokens": 1048
  }
  }

The model starts with the "right" output, but then it produces garbage.

I am running the TGI backend.

Thx!

[WaterKnight1998]

I am seeing issues also with the output of the model, but I am not using the chat-ui repo. I am just using the model in TGI. Did you find something??

[schauppi]

Nah I didn´t find something working atm..

[WaterKnight1998]

I got better results with prompt from here: https://huggingface.co/spaces/codellama/codellama-13b-chat/blob/aecd012f33bd6f8c470f3b7a462d7b30c3b65471/model.py#L25

[schauppi]

need to try this one - thx!

[nsarrazin]

Could you try this config?

    "userMessageToken": "",
    "userMessageEndToken": " [/INST] ",
    "assistantMessageToken": "",
    "assistantMessageEndToken": " </s><s>[INST] ",
    "preprompt": " ",
    "chatPromptTemplate" : "<s>[INST] <<SYS>>\n{{preprompt}}\n<</SYS>>\n\n{{#each messages}}{{#ifUser}}{{content}} [/INST] {{/ifUser}}{{#ifAssistant}}{{content}} </s><s>[INST] {{/ifAssistant}}{{/each}}",

[schauppi]

I tried it but the models responds not in that way as the hosted model on the HF chat.

I tried this prompt:

Write a python function to generate the fibonacci series.

This is the output from the model on the hosted HuggingChat Interface:

Here is an example of how you can write a Python function to generate the Fibonacci sequence:

def fibonacci(n):
    if n <= 1:
        return n
    else:
        return fibonacci(n-1) + fibonacci(n-2)

This function uses recursion to calculate each term in the Fibonacci sequence based on the previous two terms. The if statement checks whether the input value n is less than or equal to 1, and returns n directly if it is. Otherwise, it calls itself with n-1 and n-2, adds the results together, and then returns that sum.

This is the output from my local hosted model on TGI with the config you provided:

Here is a Python function that generates the Fibonacci series:

def fib(n):
if n <= 1: return n # Base case, return input value
else: # Recursive case, calculate and return the next value in the sequence
return fib(n-2)) + fib(n-1))) # The next value in the sequence is equal to the sum of the previous two values in the sequence

[mbuet2ner]

For me the model does work as expected when I use the aforementioned config. I am using a quantied GPTQ model from the bloke. However, the config does not work for follow up questions (as in HuggingChat). The previous conversation seems not to be embedded correctly in the context for the next query... Has someone found a solution to this?

[schauppi]

Which model are u using?

[spew]

Hi @nsarrazin that config is working very well for me.