how to inference

[gmdl007]

i tried to train with similar data, but i dont get the correct output, do you have example how are do the inference on your finished gguf for the the chemistry as example output? thanks a lot Guan

[YuanTony]

Would be curious to see your steps and results.

But, you can also download a finetuned GGUF file and use the commands to run inference on it. The inference command is in the instructions. You just need to replace the GGUF file name with the one you just downloaded.

https://huggingface.co/juntaoyuan/chemistry-assistant-7b/resolve/main/llama-2-7b-chat-q5_k_m.gguf?download=true

Hope this helps Tony

On Sat, May 11, 2024, 6:14 AM gmdl007 @.***> wrote:

i tried to train with similar data, but i dont get the correct output, do you have example how are do the inference on your finished gguf for the the chemistry as example output? thanks a lot Guan

— Reply to this email directly, view it on GitHub https://github.com/YuanTony/chemistry-assistant/issues/7, or unsubscribe https://github.com/notifications/unsubscribe-auth/BC5WXOTGYNXA42QAYS3G75DZBX4TDAVCNFSM6AAAAABHR3CDV6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGI4TAOBUGQZDINI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[gmdl007]

Hi Tony,

Can you point me the link where I can find the exact command which I can pass to the llama.cpp cli to do the inference?

I was using a different llm to train on my DS, and pass in the f16 base gguf + prompt to do the inference:

./../main -m open-llama-3b-v2-q4_1.gguf --lora lora.bin --lora-base open-llama-3b-v2-f16.gguf --prompt “are you compliant with xyz”

I have a dataset which is compliancy question and answer, the answer is quick simple, it is just fully compliant, partially compliant, or none compliant. The model seems learn to generate the compliancy statement, but it is never accurate.

BR Guan

From: Tony Yuan @.> Date: Tuesday, May 14, 2024 at 4:18 AM To: YuanTony/chemistry-assistant @.> Cc: gmdl007 @.>, Author @.> Subject: Re: [YuanTony/chemistry-assistant] how to inference (Issue #7) Would be curious to see your steps and results.

But, you can also download a finetuned GGUF file and use the commands to run inference on it. The inference command is in the instructions. You just need to replace the GGUF file name with the one you just downloaded.

https://huggingface.co/juntaoyuan/chemistry-assistant-7b/resolve/main/llama-2-7b-chat-q5_k_m.gguf?download=true

Hope this helps Tony

On Sat, May 11, 2024, 6:14 AM gmdl007 @.***> wrote:

i tried to train with similar data, but i dont get the correct output, do you have example how are do the inference on your finished gguf for the the chemistry as example output? thanks a lot Guan

— Reply to this email directly, view it on GitHub https://github.com/YuanTony/chemistry-assistant/issues/7, or unsubscribe https://github.com/notifications/unsubscribe-auth/BC5WXOTGYNXA42QAYS3G75DZBX4TDAVCNFSM6AAAAABHR3CDV6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGI4TAOBUGQZDINI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHubhttps://github.com/YuanTony/chemistry-assistant/issues/7#issuecomment-2109149016, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AG2FHXHFM6MD7G5IR6WIQITZCFX5RAVCNFSM6AAAAABHR3CDV6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBZGE2DSMBRGY. You are receiving this because you authored the thread.Message ID: @.***>