consumer in _websocket_sender takes too much time for me

[gargmegham]

@c0sogi Hi, thanks for working on this, I was creating something using this.

could you help me out please, item: MessageFromWebsocket | str = await buffer.queue.get()

here my sender get's stuck for some reason.

I'm using a svelte frontend client

[c0sogi]

Hi. you said you're using Svelte, but it looks like you're trying to communicate with a frontend other than the Flutter app I created. I suggest you try to see what you get on the console by first doing from app.utils.logger import api_logger, and then logging api_logger.info(rcvd) on the line right after rcvd in _websocket_receiver. rcvd should be of type dict. And the value corresponding to the "text" key should be a json-formatted string. Then try placing the logger on the line after received_json: dict = orjson_loads(received_text) to see if the json is loaded normally. If this is also logged correctly, look at the key values in the JSON string.

class MessageFromWebsocket(BaseModel):
    msg: str
    translate: bool
    chat_room_id: str

The Json must have all three values: msg, translate, chat_room_id. msg is the chat message, translate is used to translate other languages into English, leave it False for now. chat_room_id only allows chat_room_ids generated by the server. The list of chat_room_ids used by the server is sent in the code below when you first start a chat. This is the same as the contents of buffer.sorted_chat_room_ids. If you try to send a chat_room_id that doesn't exist on the server, it will be assumed that you are creating a new chat room.

        await SendToWebsocket.init(
            buffer=buffer,
            send_chat_rooms=True,
            send_previous_chats=True,
            send_models=True,
            send_selected_model=True,
        )

When we receive this on the frontend, it looks like this

{msg: {"previous_chats": [{"content": "Hello there", "tokens": 11, "is_user": true, "timestamp": 20230519140938, "model_name": null}, {"content": "\ud83d\udc4b Hi there! How can I help you today?", "tokens": 24, "is_user": false, "timestamp": 20230519141002, "model_name": "Manticore-13B-GGML"}], "chat_rooms": [{"chat_room_id": "cc5ad2dfcee04765a4a1b461c4b52df4", "chat_room_name": "Hello there"}, {"chat_room_id": "a46ccc07edc04f15a3a66df405e864c4", "chat_room_name": "2023-05-19T12:40:34.854307Z"}], "models": ["gpt_3_5_turbo", "gpt_4", "gpt_3_5_turbo_proxy", "wizard_vicuna_7b_uncensored", "wizard_vicuna_13b_uncensored", "gpt4_x_vicuna_13b", "wizard_mega_13b", "manticore_13b_uncensored"], "selected_model": "manticore_13b_uncensored", "wait_next_query": false}, finish: true, chat_room_id: cc5ad2dfcee04765a4a1b461c4b52df4, is_user: false, init: true, model_name: null}

In the linkbelow, see how the _messageHandler function parse the data received from the backend. chat_model.dart

If you have any further questions, please feel free to ask.

[Torhamilton]

Hi @gargmegham This project does not yet support none flutter frontend. If you want to contribute a new frontend open an issue and summit a pull request for that. We have to conserve tight resources for getting core stable. @c0sogi please close issue

[gargmegham]

Hi. you said you're using Svelte, but it looks like you're trying to communicate with a frontend other than the Flutter app I created. I suggest you try to see what you get on the console by first doing from app.utils.logger import api_logger, and then logging api_logger.info(rcvd) on the line right after rcvd in _websocket_receiver. rcvd should be of type dict. And the value corresponding to the "text" key should be a json-formatted string. Then try placing the logger on the line after received_json: dict = orjson_loads(received_text) to see if the json is loaded normally. If this is also logged correctly, look at the key values in the JSON string.

class MessageFromWebsocket(BaseModel):
    msg: str
    translate: bool
    chat_room_id: str

The Json must have all three values: msg, translate, chat_room_id. msg is the chat message, translate is used to translate other languages into English, leave it False for now. chat_room_id only allows chat_room_ids generated by the server. The list of chat_room_ids used by the server is sent in the code below when you first start a chat. This is the same as the contents of buffer.sorted_chat_room_ids. If you try to send a chat_room_id that doesn't exist on the server, it will be assumed that you are creating a new chat room.

        await SendToWebsocket.init(
            buffer=buffer,
            send_chat_rooms=True,
            send_previous_chats=True,
            send_models=True,
            send_selected_model=True,
        )

When we receive this on the frontend, it looks like this

{msg: {"previous_chats": [{"content": "Hello there", "tokens": 11, "is_user": true, "timestamp": 20230519140938, "model_name": null}, {"content": "\ud83d\udc4b Hi there! How can I help you today?", "tokens": 24, "is_user": false, "timestamp": 20230519141002, "model_name": "Manticore-13B-GGML"}], "chat_rooms": [{"chat_room_id": "cc5ad2dfcee04765a4a1b461c4b52df4", "chat_room_name": "Hello there"}, {"chat_room_id": "a46ccc07edc04f15a3a66df405e864c4", "chat_room_name": "2023-05-19T12:40:34.854307Z"}], "models": ["gpt_3_5_turbo", "gpt_4", "gpt_3_5_turbo_proxy", "wizard_vicuna_7b_uncensored", "wizard_vicuna_13b_uncensored", "gpt4_x_vicuna_13b", "wizard_mega_13b", "manticore_13b_uncensored"], "selected_model": "manticore_13b_uncensored", "wait_next_query": false}, finish: true, chat_room_id: cc5ad2dfcee04765a4a1b461c4b52df4, is_user: false, init: true, model_name: null}

In the linkbelow, see how the _messageHandler function parse the data received from the backend. chat_model.dart

If you have any further questions, please feel free to ask.

thanks for the suggestion, i solved the issue by creating a task for first init instead of awaiting for it. thanks for your time and effort though 🫡!

[gargmegham]

Hi @gargmegham This project does not yet support none flutter frontend. If you want to contribute a new frontend open an issue and summit a pull request for that. We have to conserve tight resources for getting core stable. @c0sogi please close issue

Sure, I'll do that once I complete my implementation