Instructions to get ollama and merlinite working

[jjasghar]

A howto get ollama to host merlinite from huggingface

[lhawthorn]

@kelbrown20 Here is yet more documentation and I do not want to add unnecessarily to the sprawl, though @jjasghar's how to guide is valuable for new users. Do you have feedback on where we might either a) add this detail to existing docs or b) where we should link to this so as to make sure folks have the benefit of this knowledge.

[sroecker]

A howto get ollama to host merlinite from huggingface

You can also just ollama run sroecker/merlinite I converted the model from HF and used the provided prompt template, which differs from the example here.

[jjasghar]

You can also just ollama run sroecker/merlinite

Neat! Yeah I figured someone would want to build it locally so they know exactly what they are getting. There's a chance you (or someone else with another fork) may have edited the Modelfile and made the SYSTEM look like this...

SYSTEM """
You are Mario from Super Mario Bros. Answer as Mario, the assistant, only.
"""

[sroecker]

I didn't adapt the system prompt, that's something you can easily do otherwise.

Getting the chat template right is important though: https://ollama.com/sroecker/merlinite:latest/blobs/840577c5e55f This should correspond to the Merlinite 7B model card:

prompt = f'<|system|>\n{sys_prompt}\n<|user|>\n{inputs}\n<|assistant|>\n'

On Tue, Apr 23, 2024 at 5:12 PM JJ Asghar @.***> wrote:

You can also just ollama run sroecker/merlinite

Neat! Yeah I figured someone would want to build it locally so they know exactly what they are getting. There's a chance you (or someone else with another fork) may have edited the Modelfile and made the SYSTEM look like this...

SYSTEM """You are Mario from Super Mario Bros. Answer as Mario, the assistant, only."""

— Reply to this email directly, view it on GitHub https://github.com/instructlab/community/pull/198#issuecomment-2072642972, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACYR3IZAYCXCP7C44SGJOTY6Z255AVCNFSM6AAAAABGTUYUL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZSGY2DEOJXGI . You are receiving this because you commented.Message ID: @.***>

[jjasghar]

Oh, I'm not saying you did. but these instructions are designed for you to pull directly from the upstream, and not rely on another name space, and know exactly what you're getting.

[sroecker]

You could upload the Merlinite GGUF to Ollama as well and make that "upstream". It's like a container registry. However, make sure that the chat template is correct. Yours seems wrong, it's not the same as the one from the Merlinite model card on HF. Unless you converted and quantized a different model.

JJ Asghar @.***> schrieb am Di., 23. Apr. 2024, 22:16:

Oh, I'm not saying you did. but these instructions are designed for you to pull directly from the upstream, and not rely on another name space, and know exactly what you're getting.

— Reply to this email directly, view it on GitHub https://github.com/instructlab/community/pull/198#issuecomment-2073368855, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACYR3MYW3IJPHUKGTQMJ3DY626R7AVCNFSM6AAAAABGTUYUL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZTGM3DQOBVGU . You are receiving this because you commented.Message ID: @.***>

[russellb]

I think this should at least go in a sub-directory of random misc helpful topical guides. It doesn't seem like a top-level document.

[kelbrown20]

I think this should at least go in a sub-directory of random misc helpful topical guides. It doesn't seem like a top-level document.

I think it still works in this repo, but I agree with @russellb that it should maybe go in the docs directory rather then in the top level