!pic failed reading parameters from request with LocalAi & stablediffusion

[35develr]

on LocalAi stablediffusion is running and working.

matrix_chatgpt_bot Config: (XXXXed out my credentials)

{
    "homeserver": "https://XXXXXXXX",
    "user_id": "@XXXXXXXXXXX",
    "password": "MistralChatBot",
    "access_token": "XXXXXXXXXXXXXXXXXXXXX",
    "device_id": "MatrixChatGPTBot",
    "room_id": "XXXXXXXXXXXXXXXXXXX",
    "openai_api_key": "xxxxxxxxxxxxxxxxxxxxxxxx",
    "gpt_api_endpoint": "http://10.XXXXXXX:8080/v1/chat/completions",
    "gpt_model": "gpt-4",
    "max_tokens": 4000,
    "top_p": 1.0,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0,
    "reply_count": 1,
    "temperature": 0.8,
    "system_prompt": "Antworte in einer Konversation auf Deutsch.",
    "image_generation_endpoint": "http://10.XXXXXXX:8080/v1/images/generations",
    "image_generation_backend": "openai",
    "timeout": 560.0
}

Log:

2023-12-20 15:14:37,788 - INFO - matrix chatgpt bot start.....

2023-12-20 15:14:40,174 - INFO - Successfully login via password

2023-12-20 15:14:46,582 - INFO - Message received in room AI

MaxR | !pic a white cat on a table

2023-12-20 15:14:46,624 - pic - ERROR - 500 Internal Server Error {"error":{"code":500,"message":"failed reading parameters from request:failed parsing request body: json: cannot unmarshal string into Go struct field OpenAIRequest.response_format of type schema.ChatCompletionResponseFormat","type":""}}

Traceback (most recent call last):

  File "/app/src/bot.py", line 1336, in pic

    b64_datas = await imagegen.get_images(

                ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/app/src/imagegen.py", line 34, in get_images

    raise Exception(

Exception: 500 Internal Server Error {"error":{"code":500,"message":"failed reading parameters from request:failed parsing request body: json: cannot unmarshal string into Go struct field OpenAIRequest.response_format of type schema.ChatCompletionResponseFormat","type":""}}

2023-12-20 15:14:46,624 - ERROR - 500 Internal Server Error {"error":{"code":500,"message":"failed reading parameters from request:failed parsing request body: json: cannot unmarshal string into Go struct field OpenAIRequest.response_format of type schema.ChatCompletionResponseFormat","type":""}}

Traceback (most recent call last):

  File "/app/src/bot.py", line 1336, in pic

    b64_datas = await imagegen.get_images(

                ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/app/src/imagegen.py", line 34, in get_images

    raise Exception(

Exception: 500 Internal Server Error {"error":{"code":500,"message":"failed reading parameters from request:failed parsing request body: json: cannot unmarshal string into Go struct field OpenAIRequest.response_format of type schema.ChatCompletionResponseFormat","type":""}}

2023-12-20 15:14:47,007 - INFO - Message received in room AI

mistral | > <@XXXXXXXXXXXXX> !pic a white cat on a table

Image generation failed

[hibobmaster]

~Can you provide debug message from localai as well?~

Edit: https://github.com/nextcloud/integration_openai/issues/65 It seems there are some breaking changes in localai which make it not fully compatible with openai api endpoint.

[hibobmaster]

https://github.com/mudler/LocalAI/blob/23eced1644695ad4de7306eee63521b4220a801c/api/schema/openai.go#L92 Can you try by removing the following line? https://github.com/hibobmaster/matrix_chatgpt_bot/blob/f4d7b9212a1256c76335eab0900bc839eb6f6ee9/src/imagegen.py#L22

There is no parameter "n" in localai v2.

[35develr]

My Error is not related to "n" but to OpenAIRequest.response_format i tried to remove the line 24 "response_format": "b64_json", then: docker compose up -d --build --force-recreate but this crashed my docker image ;-/ The other thing is: stablediffusion in localai gives me *.png the response is looking like this:

{"created":1703144083,"id":"0a7b85d9-45b2-4211-a361-5e3f3a598cc3","data":[{"embedding":null,"index":0,"url":"http://localhost:8080/generated-images/b642730922674.png"}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

all i get is the link to the url of the image.

[hibobmaster]

I will solve this maybe in tomorrow.

[hibobmaster]

v1.5.0 solve this problem, use "image_generation_backend": "localai".

https://github.com/hibobmaster/matrix_chatgpt_bot/commit/fac14a42447ae681996efb8222fdd9a06a1370bc