lbeurerkellner
commented
4 months ago I have not tested this myself, but I am open to input on experiments here. As with all mock implementations of the OpenAI API format, LMQL actually needs a very faithful re-implementation of the original API (including full support for logit biasing and prompt echoing among other things), to work in this way.
Unfortunately, my experience is that most simple wrappers do not implement the OAI API up to that level of functionality. It would be great to see the outcome of some experiments, up to what degree the API is replicated by LMStudio.
Once this is clear, you should be able to just use the endpoint via lmql.model("openai/text-davinci-003", endpoint="localhost:3000")
Can't get to uderstand how to use with LMStudio. GPT4 and I rode the documentation but both failed. This very same script without LMQL implementation is succesfull.
Anyone would have an idea ?
`import openai import lmql import asyncio
import re
a = 12
Configuration for OpenAI API
class OpenAIConfig: def init(self): self.base_url = "http://localhost:1234/v1" self.api_type = "open_ai" self.api_key = "not-needed" model = lmql.model("gpt2", endpoint="localhost:1234/v1")
Function to read file content.
This setup allows your script to dynamically
read the system message from the system_message.txt file, making it easy to update
the system message without changing the script's code.
def read_file_content(file_path): try: with open(file_path, "r") as file: return file.read().strip() except FileNotFoundError: print(f"File not found: {file_path}") return None
Function to initiate conversation with the local-model and establishes roles and where the instructions come from.
async def initiate_conversation(input_text, system_message): @lmql.query async def query(): '''lmql
access 'a' from the global namespace
async def main():
Instantiate configuration
if name == "main": asyncio.run(main())
`
outputs :
` SyntaxWarning: invalid escape sequence '\d' '''lmql User: sdkfjsfsd