Open mrseanryan opened 11 months ago
Works well enough (see local-llm-q folder) ...
Possibly setting Temp = 1 (not 0.5) adds more details to the DOT output ....
Higher temp does help with the DOT -> natural language inference (more text is generated).
Quantized versions are here:
https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF
Q4_K_M
https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF/blob/main/mistral-7b-instruct-v0.1.Q4_K_M.gguf
see https://github.com/mrseanryan/gpt-workflow/tree/master/local-llm-q