[Bug]: 503 The model is overloaded. Please try again later.

[LewisVille-flow]

Your Envirionment

가능하다면 관련된 library dependancy를 기술해주세요.

Description

발견한 Bug가 무엇인지 명확하고 구체적인 설명을 작성해주세요. 가능하다면, 해당 bug 를 reproduce 할 수 있도록 minial example code를 첨부하여 주세요. 코드가 너무 길다면, 다음 링크로 public link를 생성하세요: https://gist.github.com.

Found this error, 503 The model is overloaded. Please try again later. This error rarely occurs but it immediately cause the socket disconnection, which could lose user's input and chat history.

This error is mainly caused by google.generativeai, and this is called with chatbot for search, because of this line below, first: https://github.com/lingo-chat/prompt/blob/9774ca2fb37c267be394dcf442d4833ae9134961/Lingo_Chat/ai_server/graph/utils.py#L10

and second: https://github.com/lingo-chat/prompt/blob/9774ca2fb37c267be394dcf442d4833ae9134961/Lingo_Chat/ai_server/ws/handler.py#L51-L52

The first code(gemini llm occurs 503 error response, and then consequently handler raise the error. so, the valid approach to fix this bug should be retrying the calling gemini api.

etc(optional)

기타 자세한 Description, 관련된 Link 등이 있다면 작성해 주세요.

See https://stackoverflow.com/questions/78154047/encountering-503-error-when-calling-gemini-api-from-google-colab

[LewisVille-flow]

Maybe 'retry' could not fix this bug, already ChatGoogleGenerativeAI has the default value for the max_retries: https://api.python.langchain.com/en/latest/chat_models/langchain_google_genai.chat_models.ChatGoogleGenerativeAI.html#langchain_google_genai.chat_models.ChatGoogleGenerativeAI.max_retries

[LewisVille-flow]

May 677965fcommit fix this issue, assuming one retry could fix the model overload issue. If it happens again, I would re open this issue.