Calling chat generator externally.

[chrisk414]

I have an external app that I would like to call text-generator similar to OpenAI rest api. The protocol is based on tcp and I cannot use Rest api.

What I would like to do is to call an api that start the generation using the currently selected model and settings, something like generate_chat()

I've looked that extentions/example/script.py and it has output_modifier() callback function that I can use to hookup to relay the output back to my external app.

But there doesn't seem any api that I can start the generation, or am I missing something?

I've also looked at extensions/openai folder but it looks like it uses its own method to start the generation.

Is there a simpler way to start the generation?

thank you very much.

[sandyis]

Hello @chrisk414

not sure what you mean by "there doesn't seem any api that I can start the generation" if you are able to open this https://127.0.0.1:5000/docs then below API call should work and give you result back from LLM link to full API doc page below, make sure process is started with -api flag

curl http://127.0.0.1:5000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "This is a cake recipe:\n\n1.",
    "max_tokens": 200,
    "temperature": 1,
    "top_p": 0.9,
    "seed": 10
  }'

https://github.com/oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API

[MercyfulKing]

I am looking for instant generation as well...trying to answer live youtube chat but the issue is it doesn't load the prompt and generate a response on its own. This is a custom extension I'm trying to make - get the chat message, input the message, send reply. sounds easy right? I have no experience by the way.

import gradio as gr import pytchat

Function to get live chat messages

def get_live_chat_messages(video_id): chat = pytchat.create(video_id=video_id) while chat.is_alive(): for c in chat.get().sync_items():

Process each chat message (customize as needed)

        print(f"{c.datetime} [{c.author.name}]- {c.message}")
        # Add logic for chat bot reply here (using c.message as input)
        reply = generate_chat_response(c.message)
        print("Bot Reply:", reply)
return reply

Function to generate chat bot response

def generate_chat_response(user_input):

Add your custom logic for generating a response based on user input

# This could involve using external models or rule-based systems
response = f"Bot's Response to: {user_input}"
return response

Gradio UI setup

def ui(): with gr.Accordion("YouTube Chat Integration", open=True): with gr.Row(): video_id_input = gr.Textbox(label='YouTube Video ID') with gr.Row(): with gr.Accordion("Settings", open=False): toggle_button = gr.Checkbox(label='On/Off', value=False)

Start Gradio UI

iface = gr.Interface(fn=get_live_chat_messages, inputs="text", outputs="text")

iface.launch()

[chrisk414]

Hello @chrisk414

not sure what you mean by "there doesn't seem any api that I can start the generation" if you are able to open this https://127.0.0.1:5000/docs then below API call should work and give you result back from LLM link to full API doc page below, make sure process is started with -api flag

curl http://127.0.0.1:5000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "This is a cake recipe:\n\n1.",
    "max_tokens": 200,
    "temperature": 1,
    "top_p": 0.9,
    "seed": 10
  }'

https://github.com/oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API

Yeah, your example is using OpenAI Rest API. But I'm trying to do is to make a custom extension that can start the generation through Python function. Perhaps I didn't explain it in details but basically, the following is what I need.

generate_chat_reply("Hi there!"), to start generation with the current settings and I'll get the reply through the callback, output_modifier(). A simple example I can start the generation through a Python function would be appreciated.

Thanks.

[chrisk414]

I'm surprised to learn that there is no easy way to directly control gradio component, for example, programatically enter user input to Textbox and call "submit" event. Gradio is easy to start but it quickly runs into the limitations. I think Streamlit is much better in this case. It's more intuitive to use and allows more finer control. BTW, is there any project that's similar to this project that uses Streamlit? I strongly prefer Streamlit over Gradio, and it looks like there are a lot more activities over there, I think. In the meantime, I'm just going to implement my extension similar to "openai", oh well.

[MercyfulKing]

hate to say it but lm studio is a bit easier to interact with since it uses real openai api references to access the local llm. there is even example script you can just run. Started fooling around last night and was able to create a chatbot with python script easily that uses the openai compatible api.

Example: reuse your existing OpenAI setup

from openai import OpenAI

Point to the local server

client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-needed")

completion = client.chat.completions.create( model="local-model", # this field is currently unused messages=[ {"role": "system", "content": "Always answer in rhymes."}, {"role": "user", "content": "Introduce yourself."} ], temperature=0.7, )

[github-actions[bot]]

This issue has been closed due to inactivity for 2 months. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.