search

nextcloud / context_chat_backend

GNU Affero General Public License v3.0
4 stars 5 forks source link

Installing context chat backend on Nexcloud AIO 8.1.0V CPU-only system gets stuck on 50%. #34

Closed KyTDK closed 3 months ago

KyTDK commented 3 months ago

Describe the issue Installing context chat backend on Nexcloud AIO 8.1.0V CPU-only system gets stuck on 50%. Then it timeouts with the following error:

[app_api] Error: ExApp context_chat_backend initialization failed. Error: ExApp context_chat_backend initialization timed out (2400m)
    from ? by -- at Apr 8, 2024, 12:59:42 AM

Logs for context chat backend

``` App config: { "debug": true, "disable_aaa": false, "httpx_verify_ssl": true, "use_colors": true, "uvicorn_workers": 1, "disable_custom_model_download": false, "model_download_uri": "https://download.nextcloud.com/server/apps/context_chat_backend", "vectordb": [ "chroma", { "is_persistent": true } ], "embedding": [ "instructor", { "model_name": "hkunlp/instructor-base", "model_kwargs": { "device": "cuda" } } ], "llm": [ "llama", { "model_path": "dolphin-2.2.1-mistral-7b.Q5_K_M.gguf", "n_batch": 10, "n_ctx": 4096, "n_gpu_layers": -1, "template": "<|im_start|> system \nYou're an AI assistant good at finding relevant context from documents to answer questions provided by the user. <|im_end|>\n<|im_start|> user\nUse the following documents as context to answer the question at the end. REMEMBER to excersice source critisicm as the documents are returned by a search provider that can return unrelated documents.\n\nSTART OF CONTEXT: \n{context} \n\nEND OF CONTEXT!\n\nIf you don't know the answer or are unsure, just say that you don't know, don't try to make up an answer. Don't mention the context in your answer but rather just answer the question directly. \nQuestion: {question} Let's think this step-by-step. \n<|im_end|>\n<|im_start|> assistant\n", "end_separator": "<|im_end|>", "model_kwargs": { "device": "cuda" } } ] } App disabled at startup INFO: Started server process [1] INFO: Waiting for application startup. TRACE: ASGI [1] Started scope={'type': 'lifespan', 'asgi': {'version': '3.0', 'spec_version': '2.0'}, 'state': {}} TRACE: ASGI [1] Receive {'type': 'lifespan.startup'} TRACE: ASGI [1] Send {'type': 'lifespan.startup.complete'} INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:23000 (Press CTRL+C to quit) TRACE: 172.22.0.10:57670 - HTTP connection made TRACE: 172.22.0.10:57670 - ASGI [2] Started scope={'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('172.22.0.13', 23000), 'client': ('172.22.0.10', 57670), 'scheme': 'http', 'method': 'GET', 'root_path': '', 'path': '/heartbeat', 'raw_path': b'/heartbeat', 'query_string': b'', 'headers': '<...>', 'state': {}} TRACE: 172.22.0.10:57670 - ASGI [2] Send {'type': 'http.response.start', 'status': 200, 'headers': '<...>'} heartbeat_handler: result=ok INFO: 172.22.0.10:57670 - "GET /heartbeat HTTP/1.1" 200 OK TRACE: 172.22.0.10:57670 - ASGI [2] Send {'type': 'http.response.body', 'body': '<15 bytes>'} TRACE: 172.22.0.10:57670 - ASGI [2] Completed TRACE: 172.22.0.10:57670 - ASGI [3] Started scope={'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('172.22.0.13', 23000), 'client': ('172.22.0.10', 57670), 'scheme': 'http', 'method': 'POST', 'root_path': '', 'path': '/init', 'raw_path': b'/init', 'query_string': b'', 'headers': '<...>', 'state': {}} TRACE: 172.22.0.10:57670 - ASGI [3] Send {'type': 'http.response.start', 'status': 200, 'headers': '<...>'} INFO: 172.22.0.10:57670 - "POST /init HTTP/1.1" 200 OK TRACE: 172.22.0.10:57670 - ASGI [3] Send {'type': 'http.response.body', 'body': '<2 bytes>'} TRACE: 172.22.0.10:57670 - HTTP connection lost ```

Docker inspect for ghcr.io/nextcloud/context_chat_backend:2.0.1

``` [ { "Id": "a22b3b4cd69c5f8ce44897c71ad6bbf40e281e9eb2a90283cf27c0d589b7e16e", "Created": "2024-04-07T14:03:44.230121511Z", "Path": "python3", "Args": [ "main.py" ], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 193693, "ExitCode": 0, "Error": "", "StartedAt": "2024-04-07T14:03:45.229610554Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:d2d5792e4a189569e5315510eac46825000153e7d134c4d8c5c60b18fe822157", "ResolvConfPath": "/var/lib/docker/containers/a22b3b4cd69c5f8ce44897c71ad6bbf40e281e9eb2a90283cf27c0d589b7e16e/resolv.conf", "HostnamePath": "/var/lib/docker/containers/a22b3b4cd69c5f8ce44897c71ad6bbf40e281e9eb2a90283cf27c0d589b7e16e/hostname", "HostsPath": "/var/lib/docker/containers/a22b3b4cd69c5f8ce44897c71ad6bbf40e281e9eb2a90283cf27c0d589b7e16e/hosts", "LogPath": "/var/lib/docker/containers/a22b3b4cd69c5f8ce44897c71ad6bbf40e281e9eb2a90283cf27c0d589b7e16e/a22b3b4cd69c5f8ce44897c71ad6bbf40e281e9eb2a90283cf27c0d589b7e16e-json.log", "Name": "/nc_app_context_chat_backend", "RestartCount": 0, "Driver": "overlay2", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "docker-default", "ExecIDs": null, "HostConfig": { "Binds": null, "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "nextcloud-aio", "PortBindings": null, "RestartPolicy": { "Name": "unless-stopped", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "ConsoleSize": [ 0, 0 ], "CapAdd": null, "CapDrop": null, "CgroupnsMode": "private", "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": null, "GroupAdd": null, "IpcMode": "private", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": null, "DeviceCgroupRules": null, "DeviceRequests": null, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": null, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "Mounts": [ { "Type": "volume", "Source": "nc_app_context_chat_backend_data", "Target": "/nc_app_context_chat_backend_data" } ], "MaskedPaths": [ "/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware", "/sys/devices/virtual/powercap" ], "ReadonlyPaths": [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] }, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/0d42e951cf4f8b7cfd9d257d9feec1445227c71f613aaa238ee9c02449de38b6-init/diff:/var/lib/docker/overlay2/643969ed6872018a5519d562abe30ef863fe4e4147742fac3c671633e635a68f/diff:/var/lib/docker/overlay2/4df47099a967c3b6aebb54e37593ec81158eeb9a9077f036449e7298dada9ab8/diff:/var/lib/docker/overlay2/f557c0733400e1b0a103a1e70c79cf5005c30670584d60ba7b6af695d75fe878/diff:/var/lib/docker/overlay2/c540b0bd24d86ce24ffeb32a5bf324e6cb56fbd0e4dc55c0d0964f1de63d4604/diff:/var/lib/docker/overlay2/6878c48bdc00bf9e70c3cb7d4d6b88a66da79ef3b532e3075012a22b8af8b403/diff:/var/lib/docker/overlay2/404080eb63ff1987608ffe9fddb98e473ae0f325e439013c09e253482ecadc9e/diff:/var/lib/docker/overlay2/27408bae1984465f9d04643b4253dfa55d1daa10d3778a90d57d33753cf1949d/diff:/var/lib/docker/overlay2/5129f94e5f65e98d2c26f3538e99f0efcd2efd72d544998c6cd3e2aa38f39385/diff:/var/lib/docker/overlay2/e6abddd5ea9b7aaeea04ca3f94e7e3b170862e70c513a8962ecdfb6293552a6c/diff:/var/lib/docker/overlay2/c5f3f9826b3d24b4462d9e2c94f641a56719e79eae62fad4237be9d89960df4e/diff:/var/lib/docker/overlay2/c33fdd99314e149914257d96a0e7b37a8c707bae38556a0e264b5327e298a4db/diff:/var/lib/docker/overlay2/8ee1807a75a93a04ab43d9d114e7689f08332ed4e997f435a45f2862e02c49a8/diff:/var/lib/docker/overlay2/8e6af34373d3d74d7c6cd7517650da3052750b6ed2a960a3dfe5f304c4c85f93/diff:/var/lib/docker/overlay2/75d93edd1ae2dd709b770dd3d4aa634c143a30194e4f08df3a17bc173da00542/diff:/var/lib/docker/overlay2/73ab57f531637a44c485f1034e62fe5966f28e5a23acb2a2915604f3977f0dac/diff:/var/lib/docker/overlay2/d7260ac6674a73f19d40ae31e6c0681f61e59ec5b871473e5f546ebaacc6a776/diff:/var/lib/docker/overlay2/ffd7cd78ec40ca5d6b1f5cc0f8054f4cbb081943f6013c215aa7ed567c0d27ac/diff:/var/lib/docker/overlay2/db97a3372d612556c7cbe6bf418380a391e0ba5a5409d08cd1aa6d1252b994e2/diff:/var/lib/docker/overlay2/19d014eb759f9f0ce31928f806b64aaaef6c7f6e780c146149c2be3dfc1c30a6/diff:/var/lib/docker/overlay2/3fcc00e8f64355fadec315cad86148b2bb322dfeb515d7db4f1acb986db90a34/diff:/var/lib/docker/overlay2/e5df7bf6f19738689c877c3b105f338fd1f849c926513077b1b85f411c1baa7b/diff:/var/lib/docker/overlay2/2e279bf7b7156771d03b3fc72ae1aed8f97a9272c34b2c6064476b8e391ee530/diff:/var/lib/docker/overlay2/4ca4de0feef594e581f557a0bb1ee4edcd63b058bdd67663ef1f186fcb8f42d2/diff:/var/lib/docker/overlay2/bc6c28342a3b4f5d309b7c95da8df06594fd851b43a557d9cf9d3a467e03afcc/diff:/var/lib/docker/overlay2/235e626443e3c291d8f4fe3b5443e5014fe54b662c533f26a1e3de49f6ab43c6/diff:/var/lib/docker/overlay2/bdb00e67146ca5234c2e6a6a654fd20030553eccb9c7037e6c7aefcc88f9bc79/diff", "MergedDir": "/var/lib/docker/overlay2/0d42e951cf4f8b7cfd9d257d9feec1445227c71f613aaa238ee9c02449de38b6/merged", "UpperDir": "/var/lib/docker/overlay2/0d42e951cf4f8b7cfd9d257d9feec1445227c71f613aaa238ee9c02449de38b6/diff", "WorkDir": "/var/lib/docker/overlay2/0d42e951cf4f8b7cfd9d257d9feec1445227c71f613aaa238ee9c02449de38b6/work" }, "Name": "overlay2" }, "Mounts": [ { "Type": "volume", "Name": "nc_app_context_chat_backend_data", "Source": "/var/lib/docker/volumes/nc_app_context_chat_backend_data/_data", "Destination": "/nc_app_context_chat_backend_data", "Driver": "local", "Mode": "z", "RW": true, "Propagation": "" } ], "Config": { "Hostname": "context_chat_backend", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "AA_VERSION=2.4.0", "APP_SECRET=74mOFKPiEakLzeEvsJdfk4CcDdRwahmbAntc5FMmEreQtbFsDY1W6ivXg/Bf7UNh8a1JQve6vvHjN3H6Dd9CWsgpjDQk5aAII36wpwF4VujeZL16/zok9/+IcaT5csIy", "APP_ID=context_chat_backend", "APP_DISPLAY_NAME=Context Chat Backend", "APP_VERSION=2.0.1", "APP_HOST=0.0.0.0", "APP_PORT=23000", "APP_PERSISTENT_STORAGE=/nc_app_context_chat_backend_data", "NEXTCLOUD_URL=https://cloud.neomechanical.com", "PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "NVARCH=x86_64", "NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471", "NV_CUDA_CUDART_VERSION=11.8.89-1", "NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8", "CUDA_VERSION=11.8.0", "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64", "NVIDIA_VISIBLE_DEVICES=all", "NVIDIA_DRIVER_CAPABILITIES=compute,utility", "NV_CUDA_LIB_VERSION=11.8.0-1", "NV_NVTX_VERSION=11.8.86-1", "NV_LIBNPP_VERSION=11.8.0.86-1", "NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1", "NV_LIBCUSPARSE_VERSION=11.7.5.86-1", "NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8", "NV_LIBCUBLAS_VERSION=11.11.3.6-1", "NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1", "NV_LIBNCCL_PACKAGE_NAME=libnccl2", "NV_LIBNCCL_PACKAGE_VERSION=2.15.5-1", "NCCL_VERSION=2.15.5-1", "NV_LIBNCCL_PACKAGE=libnccl2=2.15.5-1+cuda11.8", "NVIDIA_PRODUCT_NAME=CUDA", "DEBIAN_FRONTEND=dialog" ], "Cmd": null, "Image": "ghcr.io/nextcloud/context_chat_backend:2.0.1", "Volumes": null, "WorkingDir": "/app", "Entrypoint": [ "python3", "main.py" ], "OnBuild": null, "Labels": { "maintainer": "NVIDIA CORPORATION ", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "ef251b7522817c6b81a29992b943a5251c677d70fd6d14e3cd2a7f0fadfd2135", "SandboxKey": "/var/run/docker/netns/ef251b752281", "Ports": {}, "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "nextcloud-aio": { "IPAMConfig": null, "Links": null, "Aliases": [ "context_chat_backend" ], "MacAddress": "02:42:ac:16:00:0d", "NetworkID": "399c4fafbfb32530f72f959e45042cf5f7c0b6d67d92f9a3b241879aa85a9b14", "EndpointID": "4977d91a16979acfc186d97b66bea22fc8ee930aa20980c5b1d4e1c9230eba74", "Gateway": "172.22.0.1", "IPAddress": "172.22.0.13", "IPPrefixLen": 16, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "DriverOpts": null, "DNSNames": [ "nc_app_context_chat_backend", "context_chat_backend", "a22b3b4cd69c" ] } } } } ] ```

Docker inspect for nextcloud/all-in-one:latest

``` [ { "Id": "ad72965cbce8715674445d3b1f611f0b6fb9e1f1811d91cfbc47d41f48993db8", "Created": "2024-04-07T09:35:44.404200642Z", "Path": "/start.sh", "Args": [], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1737, "ExitCode": 0, "Error": "", "StartedAt": "2024-04-07T09:37:55.728723799Z", "FinishedAt": "2024-04-07T09:36:10.071202507Z", "Health": { "Status": "healthy", "FailingStreak": 0, "Log": [ { "Start": "2024-04-08T00:45:40.908144392+10:00", "End": "2024-04-08T00:45:40.962759092+10:00", "ExitCode": 0, "Output": "Connection to localhost (::1) 80 port [tcp/http] succeeded!\nConnection to localhost (::1) 8000 port [tcp/*] succeeded!\nConnection to localhost (::1) 8080 port [tcp/http-alt] succeeded!\nConnection to localhost (::1) 8443 port [tcp/*] succeeded!\nConnection to localhost (::1) 9000 port [tcp/*] succeeded!\nConnection to localhost (127.0.0.1) 9876 port [tcp/*] succeeded!\n" }, { "Start": "2024-04-08T00:46:10.963955981+10:00", "End": "2024-04-08T00:46:11.024291717+10:00", "ExitCode": 0, "Output": "Connection to localhost (::1) 80 port [tcp/http] succeeded!\nConnection to localhost (::1) 8000 port [tcp/*] succeeded!\nConnection to localhost (::1) 8080 port [tcp/http-alt] succeeded!\nConnection to localhost (::1) 8443 port [tcp/*] succeeded!\nConnection to localhost (::1) 9000 port [tcp/*] succeeded!\nConnection to localhost (127.0.0.1) 9876 port [tcp/*] succeeded!\n" }, { "Start": "2024-04-08T00:46:41.025170316+10:00", "End": "2024-04-08T00:46:41.083936934+10:00", "ExitCode": 0, "Output": "Connection to localhost (::1) 80 port [tcp/http] succeeded!\nConnection to localhost (::1) 8000 port [tcp/*] succeeded!\nConnection to localhost (::1) 8080 port [tcp/http-alt] succeeded!\nConnection to localhost (::1) 8443 port [tcp/*] succeeded!\nConnection to localhost (::1) 9000 port [tcp/*] succeeded!\nConnection to localhost (127.0.0.1) 9876 port [tcp/*] succeeded!\n" }, { "Start": "2024-04-08T00:47:11.084383827+10:00", "End": "2024-04-08T00:47:11.16815676+10:00", "ExitCode": 0, "Output": "Connection to localhost (::1) 80 port [tcp/http] succeeded!\nConnection to localhost (::1) 8000 port [tcp/*] succeeded!\nConnection to localhost (::1) 8080 port [tcp/http-alt] succeeded!\nConnection to localhost (::1) 8443 port [tcp/*] succeeded!\nConnection to localhost (::1) 9000 port [tcp/*] succeeded!\nConnection to localhost (127.0.0.1) 9876 port [tcp/*] succeeded!\n" }, { "Start": "2024-04-08T00:47:41.168496332+10:00", "End": "2024-04-08T00:47:41.233386289+10:00", "ExitCode": 0, "Output": "Connection to localhost (::1) 80 port [tcp/http] succeeded!\nConnection to localhost (::1) 8000 port [tcp/*] succeeded!\nConnection to localhost (::1) 8080 port [tcp/http-alt] succeeded!\nConnection to localhost (::1) 8443 port [tcp/*] succeeded!\nConnection to localhost (::1) 9000 port [tcp/*] succeeded!\nConnection to localhost (127.0.0.1) 9876 port [tcp/*] succeeded!\n" } ] } }, "Image": "sha256:cf32daf3edd8420253f0fa7252b8896b59b30db82f66dd52591cdb4c8bbf2f68", "ResolvConfPath": "/var/lib/docker/containers/ad72965cbce8715674445d3b1f611f0b6fb9e1f1811d91cfbc47d41f48993db8/resolv.conf", "HostnamePath": "/var/lib/docker/containers/ad72965cbce8715674445d3b1f611f0b6fb9e1f1811d91cfbc47d41f48993db8/hostname", "HostsPath": "/var/lib/docker/containers/ad72965cbce8715674445d3b1f611f0b6fb9e1f1811d91cfbc47d41f48993db8/hosts", "LogPath": "/var/lib/docker/containers/ad72965cbce8715674445d3b1f611f0b6fb9e1f1811d91cfbc47d41f48993db8/ad72965cbce8715674445d3b1f611f0b6fb9e1f1811d91cfbc47d41f48993db8-json.log", "Name": "/nextcloud-aio-mastercontainer", "RestartCount": 0, "Driver": "overlay2", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "docker-default", "ExecIDs": null, "HostConfig": { "Binds": [ "nextcloud_aio_mastercontainer:/mnt/docker-aio-config", "/var/run/docker.sock:/var/run/docker.sock:ro" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "default", "PortBindings": { "8080/tcp": [ { "HostIp": "", "HostPort": "8080" } ] }, "RestartPolicy": { "Name": "always", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "ConsoleSize": [ 59, 232 ], "CapAdd": null, "CapDrop": null, "CgroupnsMode": "private", "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "private", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": [], "BlkioDeviceReadBps": [], "BlkioDeviceWriteBps": [], "BlkioDeviceReadIOps": [], "BlkioDeviceWriteIOps": [], "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DeviceRequests": null, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": null, "PidsLimit": null, "Ulimits": [], "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": [ "/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware", "/sys/devices/virtual/powercap" ], "ReadonlyPaths": [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ], "Init": true }, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/dcb9fac7a657ace22732e686a5294efcc4b908077c7fac1cb50e383716d22d92-init/diff:/var/lib/docker/overlay2/875b03a147c6a23baa026bceae67a71a1cc6bf386adc87e9364f5027ee1b4c08/diff:/var/lib/docker/overlay2/2e50106255b6f91ab4671d80c40aa3d003f22b1aba7e109319bf29af7a1d0bd6/diff:/var/lib/docker/overlay2/fa760e34d547f0c2af4084b9bc734e086d543b5041ed663ad0de27f8d220b2ad/diff:/var/lib/docker/overlay2/1f400812ba09e78e0a7d74739302013788e721a1ba41d133b2a16e24b26c1bd0/diff:/var/lib/docker/overlay2/914cdbd486746bc0b0e0aa7a0907f1c34750e3485cb8a2010343d0c10699651e/diff:/var/lib/docker/overlay2/978fd17dbe0633d3764c8a5188e1680b9d5321747c302bd7ae29b67ab6b22a28/diff:/var/lib/docker/overlay2/f318dbb2c5ca56bd14074fdbbea6a7ec30ccb5874c7b0a5293af731f31d86c1a/diff:/var/lib/docker/overlay2/f73d78163782ee9c3b11a0b5b54a04efebc3189ff377211b000435c0ec54a209/diff:/var/lib/docker/overlay2/021d2bb637d320b650b93b7ffa56a5adbb2775f2d94fe7c4faa5aa1f49c2de8e/diff:/var/lib/docker/overlay2/73eb7d6aa3fb9385cd95550e6adee3ac40492f9c2b9d970682faacdb37b9e80c/diff:/var/lib/docker/overlay2/9a4356a88b1e75902737df18886a8889436a5e680e06e10b1e489d160c4ff52b/diff:/var/lib/docker/overlay2/0fa37b9b61357fed0c4426d2f503a67c545099c597b17fc1c452781429268ab8/diff:/var/lib/docker/overlay2/adcc3e7b8b5e17337b5b55cb75833e82b6f7c25046f424cdb88bf851af89fd90/diff:/var/lib/docker/overlay2/c886e13a6d94e61a5d6f97987b535bbf1481cf0a14039229ae67744560670492/diff:/var/lib/docker/overlay2/1dc47f08728a545a746d100a0181e95f5b6ab452859290100cc518cf5dc6b2c0/diff:/var/lib/docker/overlay2/b06683fc369811dfb6f4297b6e095942fd2d715f300653161874dd7230f9da89/diff:/var/lib/docker/overlay2/ae790014d311b7127d0d278c353700e4ea8db2909cb1a5cd3c5657232575c62e/diff:/var/lib/docker/overlay2/36234d5583828312079e7871960c502e99b5e091c19331eb01f5f11292cc6fe5/diff", "MergedDir": "/var/lib/docker/overlay2/dcb9fac7a657ace22732e686a5294efcc4b908077c7fac1cb50e383716d22d92/merged", "UpperDir": "/var/lib/docker/overlay2/dcb9fac7a657ace22732e686a5294efcc4b908077c7fac1cb50e383716d22d92/diff", "WorkDir": "/var/lib/docker/overlay2/dcb9fac7a657ace22732e686a5294efcc4b908077c7fac1cb50e383716d22d92/work" }, "Name": "overlay2" }, "Mounts": [ { "Type": "volume", "Name": "nextcloud_aio_mastercontainer", "Source": "/var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data", "Destination": "/mnt/docker-aio-config", "Driver": "local", "Mode": "z", "RW": true, "Propagation": "" }, { "Type": "bind", "Source": "/var/run/docker.sock", "Destination": "/var/run/docker.sock", "Mode": "ro", "RW": false, "Propagation": "rprivate" } ], "Config": { "Hostname": "ad72965cbce8", "Domainname": "", "User": "root", "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "ExposedPorts": { "80/tcp": {}, "8080/tcp": {}, "8443/tcp": {}, "9000/tcp": {} }, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "APACHE_PORT=11000", "AIO_COMMUNITY_CONTAINERS=local-ai", "APACHE_IP_BINDING=0.0.0.0", "NEXTCLOUD_DATADIR=/mnt/sda1/nextcloud/data", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "PHPIZE_DEPS=autoconf \t\tdpkg-dev dpkg \t\tfile \t\tg++ \t\tgcc \t\tlibc-dev \t\tmake \t\tpkgconf \t\tre2c", "PHP_INI_DIR=/usr/local/etc/php", "PHP_CFLAGS=-fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64", "PHP_CPPFLAGS=-fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64", "PHP_LDFLAGS=-Wl,-O1 -pie", "GPG_KEYS=1198C0117593497A5EC5C199286AF1F9897469DC C28D937575603EB4ABB725861C0779DC5C0A9DE4 AFD8691FDAEDF03BDF6E460563F15A9B715376CA", "PHP_VERSION=8.3.4", "PHP_URL=https://www.php.net/distributions/php-8.3.4.tar.xz", "PHP_ASC_URL=https://www.php.net/distributions/php-8.3.4.tar.xz.asc", "PHP_SHA256=39a337036a546e5c28aea76cf424ac172db5156bd8a8fd85252e389409a5ba63" ], "Cmd": null, "Healthcheck": { "Test": [ "CMD-SHELL", "/healthcheck.sh" ] }, "Image": "nextcloud/all-in-one:latest", "Volumes": null, "WorkingDir": "/var/www/docker-aio", "Entrypoint": [ "/start.sh" ], "OnBuild": null, "Labels": {}, "StopSignal": "SIGQUIT" }, "NetworkSettings": { "Bridge": "", "SandboxID": "e344d68f39563d1c34ca2bae2d867f41ef4311df96a62679e2a718a5eae3c56b", "SandboxKey": "/var/run/docker/netns/e344d68f3956", "Ports": { "80/tcp": null, "8080/tcp": [ { "HostIp": "0.0.0.0", "HostPort": "8080" }, { "HostIp": "::", "HostPort": "8080" } ], "8443/tcp": null, "9000/tcp": null }, "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "5ac57d58a04e5b5f7dee84f8607b4c0837a5f87c504eb64dc2e8ce5bc42b26cc", "Gateway": "172.17.0.1", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "172.17.0.2", "IPPrefixLen": 16, "IPv6Gateway": "", "MacAddress": "02:42:ac:11:00:02", "Networks": { "bridge": { "IPAMConfig": null, "Links": null, "Aliases": null, "MacAddress": "02:42:ac:11:00:02", "NetworkID": "fab3ceea04c0efcabdb7fe1d36024381331997eb5e46152a3ded8b306f5a752b", "EndpointID": "5ac57d58a04e5b5f7dee84f8607b4c0837a5f87c504eb64dc2e8ce5bc42b26cc", "Gateway": "172.17.0.1", "IPAddress": "172.17.0.2", "IPPrefixLen": 16, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "DriverOpts": null, "DNSNames": null }, "nextcloud-aio": { "IPAMConfig": null, "Links": null, "Aliases": null, "MacAddress": "02:42:ac:16:00:0c", "NetworkID": "399c4fafbfb32530f72f959e45042cf5f7c0b6d67d92f9a3b241879aa85a9b14", "EndpointID": "9ac50810eff741c527f47b6fe39af4c126aab8c5860a264d5aad13268c4493fe", "Gateway": "172.22.0.1", "IPAddress": "172.22.0.12", "IPPrefixLen": 16, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "DriverOpts": null, "DNSNames": [ "nextcloud-aio-mastercontainer", "ad72965cbce8" ] } } } } ] ```

The config.yml of chat context backend Again, its a CPU only system so I don't know why its using cuda

``` debug: true disable_aaa: false httpx_verify_ssl: true use_colors: true uvicorn_workers: 1 # model files download configuration disable_custom_model_download: false model_download_uri: https://download.nextcloud.com/server/apps/context_chat_backend vectordb: chroma: is_persistent: true # chroma_server_host: # chroma_server_http_port: # chroma_server_ssl_enabled: # chroma_server_api_default_path: weaviate: # auth_client_secret: # url: http://localhost:8080 embedding: instructor: model_name: hkunlp/instructor-base model_kwargs: device: cuda llama: model_path: dolphin-2.2.1-mistral-7b.Q5_K_M.gguf n_batch: 16 n_ctx: 2048 hugging_face: # model_name: all-MiniLM-L6-v2 model_name: sentence-transformers/all-mpnet-base-v2 model_kwargs: device: cuda llm: llama: model_path: dolphin-2.2.1-mistral-7b.Q5_K_M.gguf n_batch: 10 n_ctx: 4096 n_gpu_layers: -1 template: "<|im_start|> system \nYou're an AI assistant good at finding relevant context from documents to answer questions provided by the user. <|im_end|>\n<|im_start|> user\nUse the following documents as context to answer the question at the end. REMEMBER to excersice source critisicm as the documents are returned by a search provider that can return unrelated documents.\n\nSTART OF CONTEXT: \n{context} \n\nEND OF CONTEXT!\n\nIf you don't know the answer or are unsure, just say that you don't know, don't try to make up an answer. Don't mention the context in your answer but rather just answer the question directly. \nQuestion: {question} Let's think this step-by-step. \n<|im_end|>\n<|im_start|> assistant\n" end_separator: <|im_end|> model_kwargs: device: cuda ctransformer: model: dolphin-2.2.1-mistral-7b.Q5_K_M.gguf template: "<|im_start|> system \nYou're an AI assistant good at finding relevant context from documents to answer questions provided by the user. <|im_end|>\n<|im_start|> user\nUse the following documents as context to answer the question at the end. REMEMBER to excersice source critisicm as the documents are returned by a search provider that can return unrelated documents.\n\nSTART OF CONTEXT: \n{context} \n\nEND OF CONTEXT!\n\nIf you don't know the answer or are unsure, just say that you don't know, don't try to make up an answer. Don't mention the context in your answer but rather just answer the question directly. \nQuestion: {question} Let's think this step-by-step. \n<|im_end|>\n<|im_start|> assistant\n" end_separator: <|im_end|> config: gpu_layers: -1 model_kwargs: device: cuda hugging_face: model_id: gpt2 task: text-generation pipeline_kwargs: config: max_length: 200 template: "" ```

Setup Details (please complete the following information):

kyteinsky commented 3 months ago

Hello, yeah this is a known issue. A fix will be available very soon. I see you've already found the issue where it is tracked so will be closing this one to keep discussions at one place.