Local Models

[DarpanGanatra]

I would like the ability to use local models as "Prompt Drivers". Simple example is using LMStudio's Local Inference Server option where I can have a model deployed to an endpoint and call it using (currently) LangChain by altering the base_path of an OpenAI call.

Here's the example client requested provided:

curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{ 
    "messages": [ 
      { "role": "user", "content": "### Instruction: Introduce yourself.\n###Response: " }
    ], 
    "stop": ["### Instruction:"],
    "temperature": 0.7, 
    "max_tokens": -1,
    "stream": false
  }'

Describe alternatives you've considered I've attempted to do this with the method described above:

import openai

# Put your URI end point:port here for your local inference server (in LM Studio)
openai.api_base = "http://localhost:1234/v1"
# Put in an empty API Key
openai.api_key = ""

agent = Agent(
    OpenAiChatPromptDriver(
        api_base=openai.api_base,
        api_key=None,
        model="test",
    )
)

But I get:

WARNING:root:<RetryCallState 11580606928: attempt #1; slept for 0.0; last result: failed (AuthenticationError No API key provided. You can set your API key in code using 'openai.api_key = <API-KEY>', or you can set the environment variable OPENAI_API_KEY=<API-KEY>). If your API key is stored in a file, you can point the openai module at it with 'openai.api_key_path = <PATH>'. You can generate API keys in the OpenAI web interface. See https://platform.openai.com/account/api-keys for details.)>

Please let me know if any other information is needed.

[collindutter]

Hey @DarpanGanatra! You can use the Hugging Face Pipelines Prompt Driver to run inference against local models. Are you looking for a Prompt Driver for LM Studio specifically?

[DarpanGanatra]

I see, I didn't look into that before I raised this, I'll close. Thank you @collindutter !

[collindutter]

Sounds good! Feel free to re-open if you face any issues with that driver 🙂

[LeventeNagy]

I know it is closed, but if anyone still interested, griptape works with LM Studio

Example

import logging

os.environ["OPENAI_API_KEY"] = "NULL"
os.environ["OPENAI_BASE_URL"] = "http://localhost:1234/v1"

Set the logging level to ERROR, this will provide a much cleaner output and will only display logs of the highest priority

agent = Agent(
    logger_level=logging.ERROR,
)

Function, passing the prompt and the agent and returns the agent output

def chat(prompt, agent):
    agent_response = agent.run(prompt)
    return agent_response.output_task.output.value

Then you can just call the function

for example (I use streamlit)

prompt = st.chat_input("Question")

if prompt:
    st.chat_message("user").write(prompt)

    with st.spinner("Thinking"):
        response = chat(prompt=prompt, agent=agent)
        st.write(response)

Without streamlit

prompt= input("Question:  ")

if prompt:
   answer =chat(prompt=prompt, agent=agent)
   print(answer)

Hope it helps