Not working with any provider!!!

sergiupapuc commented 2 months ago

Which version of integration_openai are you using?

3.1.0

Which version of Nextcloud are you using?

v30.0.0

Which browser are you using? In case you are using the phone App, specify the Android or iOS version and device please.

chrome

Describe the Bug

..have set with localai(separate server), Mistral AI, OPENAI...none work

Expected Behavior

chat with ai

To Reproduce

set localai server ...make accound mistrall ai...etc

Lumpiness commented 2 months ago

Clear installation of:

Nextcloud Hub 9 (30.0.0) from docker nextcloud:30-apache + postgres:alpine + nextcloud/aio-redis:latest + nextcloud:30-apache (as cron). Compose from git
Nextcloud Assistant: 2.0.4
OpenAI and LocalAI integration: 3.1.0
LocalAI from quay.io/go-skynet/local-ai:master-cublas-cuda12-ffmpeg

Here is the cron container log: 2024-09-19T19:20:00.363131000Z crond: USER www-data pid 35 cmd php -f /var/www/html/cron.php

Here is the LocalAI container log:

2024-09-19T19:16:20.495380000Z 10:16PM INF Success ip=172.25.0.11 latency="130.868µs" method=GET status=200 url=/v1/models
2024-09-19T19:16:20.575248000Z 10:16PM INF Success ip=172.25.0.11 latency="112.448µs" method=GET status=200 url=/v1/models
2024-09-19T19:16:29.424039000Z 10:16PM INF Success ip=127.0.0.1 latency="26.518µs" method=GET status=200 url=/readyz
2024-09-19T19:16:42.437783000Z 10:16PM INF Success ip=172.25.0.11 latency="168.455µs" method=GET status=200 url=/v1/models
2024-09-19T19:16:47.919126000Z 10:16PM INF Success ip=172.25.0.11 latency="115.223µs" method=GET status=200 url=/v1/models
2024-09-19T19:16:56.368200000Z 10:16PM INF Success ip=172.25.0.11 latency="126.811µs" method=GET status=200 url=/v1/models
2024-09-19T19:17:12.039540000Z 10:17PM INF Success ip=172.25.0.11 latency="172.081µs" method=GET status=200 url=/v1/models
2024-09-19T19:17:20.727210000Z 10:17PM INF Success ip=172.25.0.11 latency="114.512µs" method=GET status=200 url=/v1/models
2024-09-19T19:17:23.040241000Z 10:17PM INF Success ip=172.25.0.11 latency="132.72µs" method=GET status=200 url=/v1/models
2024-09-19T19:17:29.487291000Z 10:17PM INF Success ip=127.0.0.1 latency="20.102µs" method=GET status=200 url=/readyz
2024-09-19T19:18:29.555984000Z 10:18PM INF Success ip=127.0.0.1 latency="61.416µs" method=GET status=200 url=/readyz
2024-09-19T19:19:29.614084000Z 10:19PM INF Success ip=127.0.0.1 latency="47.724µs" method=GET status=200 url=/readyz
2024-09-19T19:20:02.140932000Z 10:20PM INF Success ip=172.25.0.15 latency=753.413383ms method=POST status=200 url=/v1/chat/completions
2024-09-19T19:20:02.175222000Z 10:20PM INF Loading model 'DreamShaper_8_pruned.safetensors' with backend diffusers
2024-09-19T19:20:04.118873000Z 10:20PM INF Loading model 'DreamShaper_8_pruned.safetensors' with backend diffusers
2024-09-19T19:20:06.058559000Z 10:20PM INF Loading model 'DreamShaper_8_pruned.safetensors' with backend diffusers
2024-09-19T19:20:07.974876000Z 10:20PM INF Success ip=172.25.0.15 latency=5.799845002s method=POST status=200 url=/v1/images/generations
2024-09-19T19:20:07.979430000Z 10:20PM INF Success ip=172.25.0.15 latency="92.86µs" method=GET status=200 url=/generated-images/b642726039242.png
2024-09-19T19:20:07.980475000Z 10:20PM INF Success ip=172.25.0.15 latency="46.372µs" method=GET status=200 url=/generated-images/b643240397742.png

At 19:16 I opened two browser tabs with NextCloud. In the first tab, I launched Nextcloud Assistant "Chat with AI" and entered a prompt. Then, in the second tab, I launched Nextcloud Assistant “Generate image” and entered a prompt to generate 3 images. At 19:20, a response from LocalAI appeared in both tabs. It looks like all requests are queued in a cron job queue rather than being executed immediately. But in the Nextcloud “Artificial Intelligence” settings, after entering the correct LocalAI address, models immediately appear in the combo box.

Clear installation of:

nextcloud:29-apache + ...
Nextcloud Assistant: 1.1.0
OpenAI and LocalAI integration: 2.0.3

works correctly. All requests are executed immediately.

ctft1 commented 2 months ago

I confirm this behavior, the Chat only answers once every 5 minutes (9:00 / 9:05 / 9:10 / ...) This should be fixed

jon-bit commented 1 month ago

Same. I am tring to use openai (just because it's cheap for whisper), and it takes so long to respond.

ctft1 commented 1 month ago

I found a workaround to create a worker that will process requests as soon as they arrive. It still queue them, but if you have no other request it will be processed immediately. You can also run more workers to parallel process

Just use the following command (from you NC installation, or use the "docker exec" if using Docker) and it should work

occ background-job:worker 'OC\TaskProcessing\SynchronousBackgroundJob'

Let me know if it works for you

rutgerputter commented 1 month ago

I found a workaround to create a worker that will process requests as soon as they arrive. It still queue them, but if you have no other request it will be processed immediately. You can also run more workers to parallel process

Just use the following command (from you NC installation, or use the "docker exec" if using Docker) and it should work

occ background-job:worker 'OC\TaskProcessing\SynchronousBackgroundJob'

Let me know if it works for you

This works indeed! But it's only a workaround.

Hopefully this information will point the devs in the right direction. As far as I can tell a LLM request should trigger a worker and it is not, probably the cron job triggers the same worker and that explains why these LLM requests run every 5 minutes.

Lumpiness commented 1 month ago

Running a worker occ background-job:worker 'OC\TaskProcessing\SynchronousBackgroundJob' helped me too

jon-bit commented 1 month ago

I run

sudo docker exec --user www-data -it <all-in-one ID and nextcloud-aio ID> php occ background-job:worker 'OC\TaskProcessing\SynchronousBackgroundJob'

I get

Command "background-job:worker" is not defined.  

  Did you mean one of these?                       
      background-job:delete                        
      background-job:execute                       
      background-job:list                          
      background:ajax                              
      background:cron                              
      background:webcron

ctft1 commented 1 month ago

Did you replace the all-in-one ID and nextcloud-aio ID with an number matching the user ID the AIO is using?

When running the command without the -u param, it will return something like :

Console has to be executed with the user that owns the file config/config.php
Current user id: 0
Owner id of config.php: 33

In that case the integer you should use is 33, which would be the value to be added in you command, as "-u 33"

If using the standard AIO containers from Nextcloud, you should have a command like:

sudo docker exec -it -u 33 nextcloud-aio-nextcloud /var/www/html/occ background-job:worker 'OC\TaskProcessing\SynchronousBackgroundJob'

kyteinsky commented 1 month ago

Hi, sorry for the delayed documentation, the app is working, it just scheduled cron jobs for every task due to a change in the underlying architecture (which means 5 mins delay for each task). It looks like you have already figured out the background worker, which is like an accelerator for background/cron jobs. See here for detailed docs on the setup of those workers: https://docs.nextcloud.com/server/latest/admin_manual/ai/overview.html#improve-ai-task-pickup-speed Remember to select the task processing providers in admin settings -> AI to see the difference.

@jon-bit It is available since NC 30.

d4g commented 1 month ago

How does this fit for the chat feature of nextcloud assistant? This "workaround" can't be the solution, right? Will this be changed again? Or configurable? I see why you would want to schedule that for large user bases, but 5 minutes? And the solution is to run a binary in screen?

nextcloud / integration_openai