Use cl.user_session inside Custom Providers

[Uranium2]

Hello,

How can we used the context sessions of chainlit for a custom provider? What I with to do, is to pass messages between the provider and the normal chat.

I tried to set up cl as a global variable, to pass it to my provider file, but it does not work.

Seems like it's only possible to use cl from the decorator you listed, and I can't find any decorators for a class to give the context of chainlit.

[willydouhard]

You cannot do that. user_session is only accessible within the websocket session context. The prompt playground does http requests to the chainlit server (hence not within the websocket context).

What kind of information do you need from the user_session in the custom llm provider?

[zboyles]

@Uranium2 if you can handle delivering the messages via UserSession, you can make a singleton state manager, instantiate it in on_chat_start, and then you'll be able to maintain reliable user_session access in multiple areas. This allowed me to create and use multiple derived Chat Profile template classes that all acted as expected when used with a lightweight Chat Profile template loader class to rewire the Chainlit decorators to the active profile's handler functions ie on_message, set_chat_profiles, etc. It also allowed me to write a decorator class for automatically converting a function into an InputWidget, each paired and values sync'd with user_session to allow access to the user settings from anywhere in the active custom Chat Profile templates. I'll provide the state manager code below but be aware that although we've not experienced any issues, it's only used with an internal group of trusted people, so I'd encourage you to perform more thorough testing if the initial concept satisfies your use case.

Note: My results are with v0.7.700 deployments.

# state_manager.py
class StateManager:
    _instance = None

    def __new__(cls, *args, **kwargs):
        if cls._instance is None:
            cls._instance = super(StateManager, cls).__new__(cls)
            cls._instance.initialized = False
        return cls._instance

    def init(self, user_session: UserSession):
        self.user_session = user_session
        self.initialized = True

    def get_state(self, key: str, default=None):
        if not self.initialized:
            raise Exception("StateManager not initialized")
        return self.user_session.get(key, default)

    def set_state(self, key: str, value):
        if not self.initialized:
            raise Exception("StateManager not initialized")
        self.user_session.set(key, value)
        return value

    @property
    def active_chat_profile_name(self) -> str:
        return self.get_state("chat_profile", None)

    def is_active_profile(self, chat_profile_name: str) -> bool:
        return self.active_chat_profile_name == chat_profile_name

# app.py
@cl.on_chat_start
async def on_chat_start():
    StateManager().init(cl.user_session)

    # initialize anything dependent on the user session

    # if managing multiple chat profile's, register initial profile's functions with the `Chainlit`
    # decorators because, except for `action_callback`,  only 1 of each type is registered at
    # the same time.

[willydouhard]

The new release should help with that issue. Especially the new metadata field on the User class.