search

dusty-nv / NanoLLM

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector DB, and RAG.
https://dusty-nv.github.io/NanoLLM/
MIT License
192 stars 29 forks source link

Trying to set system prompt externally with a Python client #35

Open lukecdash opened 2 months ago

lukecdash commented 2 months ago

Hi I am struggling to find out how to connect a new client to the websocket handler in web_chat.py

My end goal is to set the system prompt externally using a Python client, but I can't for the life of me find the right way to connect to the web socket in the first place.

Does a separate message handler need to be implemented in a fork? Or should I be registering a new route and listener within the actual flask app?

Or is there a Python client that you would recommend for simply sending the message straight to what is already listening?

dusty-nv commented 2 months ago

@lukecdash I need to document the websockets API and it evolved with the AgentStudio DynamicAgent/DynamicPlugin setup to be a lightweight RPC where you can call/retrieve/set any plugin's attribute or function by passing dict's over the websocket. But for now you may just want to add another REST endpoint (i.e. with Flask) with your needed functions like setting the system prompt.