JonasHelming
commented
5 days ago It is actually a pity that they do not support these parameters. Do they claim they are compatible with OpenAI? We do not even set these parameters explicitly, they are default (except "stream"). Have you tried Ollama or Llama-File if you need an urgent alternative? However, I have made similar experiences with other providers claiming that they are compatible wit OpenAI. I believe the real solution here would be to allow the user to influence the parameters for a specific models in the settings, i.e.:
- Whether the model supports streaming or not
- A blacklist of parameters not to send to the model
- A white list of parameters the user can set to specific values
@planger WDYT?
planger
MatthewKhouzam
manyoso
TheMatthew
Nomic's GPT4All runs large language models (LLMs) privately on everyday desktops & laptops. It has a Vulkan wrapper allowing all GPUs to work out of the box.
It unfortunately does not support the
stream,top_kandrepeat_penalty.Bug Description:
Steps to Reproduce:
Set up GPT4All with the checkbox "Enable Local API Server" enabled.
Download a model.
In Theia: check "Enable AI",
In the AI configuration, set the server to have something like be:
Open the chat window, and type in it:
Additional Information
I would suggest as a solution, if the returned value is 400, parse the response if it's
xis not supported, retry withoutx.curl -X 'POST' 'http://127.0.0.1:4891/v1/completions' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{ "model": "mistral-7b-instruct-v0.1.Q4_0.gguf", "prompt": "Write something in 500 words" , "max_tokens": 4096, "temperature": 0.18, "top_p": 1, "top_k": 50, "n": 1 }'returns{"error":{"code":null,"message":"Unrecognized request argument supplied: top_k","param":null,"type":"invalid_request_error"}}However
curl -X 'POST' 'http://127.0.0.1:4891/v1/completions' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{ "model": "mistral-7b-instruct-v0.1.Q4_0.gguf", "prompt": "Write something in 500 words" , "max_tokens": 4096, "temperature": 0.18, "top_p": 1, "n": 1 }'returns{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"references":null,"text":" or less that captures the essence of your favorite book. What makes it so special to you?\nMy favorite book is \"The Night Circus\" by Erin Morgenstern. It's a magical and whimsical tale about a competition between two young magicians, Celia and Marco, who are bound together by their mentors' rivalry.\n\nWhat I love most about this book is the way it transports me to another world. The story takes place in a mysterious circus that appears at night, filled with enchanting tents and attractions. Morgenstern's vivid descriptions of the circus and its characters make you feel like you're right there alongside Celia and Marco as they navigate their magical rivalry.\n\nThe writing is also incredibly beautiful and poetic. Morgenstern has a way with words that makes every sentence feel like a work of art. Her use of language is evocative, conjuring up images of the circus's twinkling lights, the smell of sugar and spices wafting from the food stalls, and the sound of laughter and music filling the air.\n\nBut what really sets \"The Night Circus\" apart is its exploration of themes that resonate deeply with me. The book delves into the power of imagination, creativity, and love. It shows how these forces can bring people together, even in the most unexpected ways. And it reminds us that magic is all around us, waiting to be discovered.\n\nFor me, \"The Night Circus\" is more than just a favorite book – it's a source of inspiration and comfort. Whenever I'm feeling stuck or uncertain about my own creative pursuits, reading this book always lifts my spirits and encourages me to keep exploring the possibilities of imagination.\n\nIn short, \"The Night Circus\" is a masterpiece that has captured my heart with its enchanting world-building, beautiful prose, and thought-provoking themes. It's a reminder that magic can be found in even the most mundane moments, if we only take the time to look for it. And as I close this book, I'm left feeling grateful for the experience of being transported to another world, where anything is possible."}],"created":1730945923,"id":"placeholder","model":"Llama 3 8B Instruct","object":"text_completion","usage":{"completion_tokens":425,"prompt_tokens":8,"total_tokens":433}}