Data Sources are Empty

[fahdmirza]

Hello, I have seen this issue being raised in other threads without any resolution, so please don't close this until resolved. I have been trying to get this working for hours and despite of setting VITE_QA_FOUNDRY_URL=http://localhost:8000 and local.metadata.yaml , the data sources are still empty. I was trying to review this tool for my channel and its quite off-putting to waste hours on this. Could I request you to fix this basic functionality of RAG before even making this project public? Thanks

[S1LV3RJ1NX]

Hello @fahdmirza , Apologies for the inconvenience, we are working on revamping and fixing the UI and backend for data sources creation. The PR is in progress and be released soon.

Also, can you check the docker-compose logs and let us know what issue you are facing, cause we are not able to re-produce on your end.

[S1LV3RJ1NX]

@fahdmirza , can you take the latest pull from main and check, your issue must have been resolved now.

[fahdmirza]

Thank you, let me try it out shortly.

[fahdmirza]

Still doesn't work .

(cognita) Ubuntu@0068-kci-prxmx10127:~/cognita$ docker compose --env-file compose.env up --build WARN[0000] /home/Ubuntu/cognita/docker-compose.yaml:versionis obsolete [+] Building 0.8s (33/33) FINISHED docker:default => [cognita-backend internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 2.36kB 0.0s => [cognita-backend internal] load metadata for docker.io/library/python:3.11 0.2s => [cognita-backend internal] load .dockerignore 0.0s => => transferring context: 307B 0.0s => [cognita-backend 1/12] FROM docker.io/library/python:3.11@sha256:01b1035a2912ade481cf6db2381dc10c97e 0.0s => [cognita-backend internal] load build context 0.2s => => transferring context: 38.68MB 0.2s => CACHED [cognita-backend 2/12] RUN apt-get update -y && apt-get install ffmpeg libsm6 libxext6 popple 0.0s => CACHED [cognita-backend 3/12] RUN python3 -m venv /virtualenvs/venv/ 0.0s => CACHED [cognita-backend 4/12] COPY backend/requirements.txt /tmp/requirements.txt 0.0s => CACHED [cognita-backend 5/12] COPY backend/vectordb.requirements.txt /tmp/vectordb.requirements.txt 0.0s => CACHED [cognita-backend 6/12] COPY backend/parsers.requirements.txt /tmp/parsers.requirements.txt 0.0s => CACHED [cognita-backend 7/12] RUN python3 -m pip install -U pip setuptools wheel && python3 -m p 0.0s => CACHED [cognita-backend 8/12] RUN if [ "0" = "1" ]; then pip install "torch==2.2.2+cu121" --extra-in 0.0s => CACHED [cognita-backend 9/12] RUN if [ "0" = "1" ]; then python3 -m pip install --use-pep517 --no-ca 0.0s => CACHED [cognita-backend 10/12] RUN if [ "0" = "1" ]; then python3 -m pip install --use-pep517 --no-ca 0.0s => CACHED [cognita-backend 11/12] COPY . /app 0.0s => CACHED [cognita-backend 12/12] WORKDIR /app 0.0s => [cognita-backend] exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:2fe61035e24141d4dbbb6cafe8621bc0eb163a8a9cba3af5ef619340db39d735 0.0s => => naming to docker.io/library/cognita-cognita-backend 0.0s => [cognita-frontend internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 915B 0.0s => [cognita-frontend internal] load metadata for docker.io/library/node:18.2.0 0.2s => [cognita-frontend internal] load .dockerignore 0.0s => => transferring context: 60B 0.0s => [cognita-frontend internal] load build context 0.0s => => transferring context: 5.45MB 0.0s => [cognita-frontend build-step 1/8] FROM docker.io/library/node:18.2.0@sha256:52bda4c171f379c1dcba5411d 0.0s => CACHED [cognita-frontend production-stage 2/4] RUN npm install -g serve 0.0s => CACHED [cognita-frontend production-stage 3/4] WORKDIR /app 0.0s => CACHED [cognita-frontend build-step 2/8] WORKDIR /build 0.0s => CACHED [cognita-frontend build-step 3/8] COPY . . 0.0s => CACHED [cognita-frontend build-step 4/8] RUN yarn install 0.0s => CACHED [cognita-frontend build-step 5/8] RUN test -e .env || cp .env.example .env 0.0s => CACHED [cognita-frontend build-step 6/8] RUN if [ -n "http://localhost:8000" ]; then export VITE_ 0.0s => CACHED [cognita-frontend build-step 7/8] RUN if [ -n "true" ]; then export VITE_USE_LOCAL=true; 0.0s => CACHED [cognita-frontend build-step 8/8] RUN yarn build 0.0s => CACHED [cognita-frontend production-stage 4/4] COPY --from=build-step /build/dist /app/dist 0.0s => [cognita-frontend] exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:de18b1264d0b93eb748887d764c975b9d9f3161eb672359c1572371e0e38eb3f 0.0s => => naming to docker.io/library/cognita-cognita-frontend 0.0s [+] Running 7/5 ✔ Network cognita_cognita-docker Created 0.1s ✔ Container qdrant Created 0.1s ✔ Container infinity Created 0.1s ✔ Container ollama Created 0.1s ✔ Container cognita-postgres Created 0.1s ✔ Container cognita-backend Created 0.0s ✔ Container cognita-frontend Created 0.0s Attaching to cognita-backend, cognita-frontend, cognita-postgres, infinity, ollama, qdrant qdrant | _ _ qdrant | __ _ __| |_ __ __ _ _ __ | |_ qdrant | / _ |/ _| '__/ _ | '_ | | qdrant | | (| | (| | | | (| | | | | | qdrant | _, |_,|| _,|| ||\| qdrant | |_| qdrant | qdrant | Version: 1.8.4, build: 984f55d6 qdrant | Access web UI at http://localhost:6333/dashboard qdrant | qdrant | 2024-06-21T01:28:11.611776Z INFO storage::content_manager::consensus::persistent: Loading raft state from ./storage/raft_state.json qdrant | 2024-06-21T01:28:11.615167Z INFO qdrant: Distributed mode disabled qdrant | 2024-06-21T01:28:11.615309Z INFO qdrant: Telemetry reporting enabled, id: 75c29589-ccc4-4348-8cb2-ac379b14cd16 qdrant | 2024-06-21T01:28:11.617276Z INFO qdrant::actix: TLS disabled for REST API qdrant | 2024-06-21T01:28:11.617482Z INFO qdrant::actix: Qdrant HTTP listening on 6333 qdrant | 2024-06-21T01:28:11.617547Z INFO actix_server::builder: Starting 5 workers qdrant | 2024-06-21T01:28:11.617628Z INFO actix_server::server: Actix runtime found; starting in Actix runtime ollama | 2024/06/21 01:28:11 routes.go:1011: INFO server config env="map[OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST: OLLAMA_KEEP_ALIVE:24h OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_VRAM:0 OLLAMA_MODELS: OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost: https://localhost: http://127.0.0.1 https://127.0.0.1 http://127.0.0.1: https://127.0.0.1: http://0.0.0.0 https://0.0.0.0 http://0.0.0.0: https://0.0.0.0: app:// file:// tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_TMPDIR:]" ollama | time=2024-06-21T01:28:11.619Z level=INFO source=images.go:740 msg="total blobs: 5" ollama | time=2024-06-21T01:28:11.619Z level=INFO source=images.go:747 msg="total unused blobs removed: 0" ollama | time=2024-06-21T01:28:11.619Z level=INFO source=routes.go:1057 msg="Listening on [::]:11434 (version 0.1.42)" ollama | time=2024-06-21T01:28:11.620Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama672448768/runners qdrant | 2024-06-21T01:28:11.621302Z INFO qdrant::tonic: Qdrant gRPC listening on 6334 qdrant | 2024-06-21T01:28:11.621319Z INFO qdrant::tonic: TLS disabled for gRPC API cognita-postgres | cognita-postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization cognita-postgres | cognita-postgres | 2024-06-21 01:28:11.737 UTC [1] LOG: starting PostgreSQL 13.15 (Debian 13.15-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit cognita-postgres | 2024-06-21 01:28:11.737 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 cognita-postgres | 2024-06-21 01:28:11.737 UTC [1] LOG: listening on IPv6 address "::", port 5432 cognita-postgres | 2024-06-21 01:28:11.740 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" cognita-postgres | 2024-06-21 01:28:11.745 UTC [27] LOG: database system was shut down at 2024-06-21 01:27:42 UTC cognita-postgres | 2024-06-21 01:28:11.753 UTC [1] LOG: database system is ready to accept connections ollama | time=2024-06-21T01:28:14.552Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 rocm_v60002]" ollama | time=2024-06-21T01:28:14.553Z level=INFO source=types.go:71 msg="inference compute" id=0 library=cpu compute="" driver=0.0 name="" total="47.1 GiB" available="4.0 GiB" infinity | INFO: Started server process [1] infinity | INFO: Waiting for application startup. infinity | INFO 2024-06-21 01:28:16,634 infinity_emb INFO: select_model.py:57 infinity | model=mixedbread-ai/mxbai-embed-large-v1 selected, infinity | using engine=torch and device=None infinity | INFO 2024-06-21 01:28:16,764 SentenceTransformer.py:189 infinity | sentence_transformers.SentenceTransformer infinity | INFO: Use pytorch device_name: cpu infinity | INFO 2024-06-21 01:28:16,766 SentenceTransformer.py:197 infinity | sentence_transformers.SentenceTransformer infinity | INFO: Load pretrained SentenceTransformer: infinity | mixedbread-ai/mxbai-embed-large-v1 infinity | /app/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True. infinity | warnings.warn( infinity | INFO 2024-06-21 01:28:18,032 infinity_emb INFO: Adding acceleration.py:30 infinity | optimizations via Huggingface optimum. infinity | The BetterTransformer implementation does not support padding during training, as the fused kernels do not support attention masks. Beware that passing padded batched data during training may result in unexpected outputs. Please refer to https://huggingface.co/docs/optimum/bettertransformer/overview for more details. infinity | /app/.venv/lib/python3.10/site-packages/optimum/bettertransformer/models/encoder_models.py:301: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. (Triggered internally at ../aten/src/ATen/NestedTensorImpl.cpp:178.) infinity | hidden_states = torch._nested_tensor_from_mask(hidden_states, ~attention_mask) infinity | INFO 2024-06-21 01:28:19,033 infinity_emb INFO: Getting select_model.py:80 infinity | timings for batch_size=8 and avg tokens per infinity | sentence=1 infinity | 0.86 ms tokenization infinity | 141.13 ms inference infinity | 0.27 ms post-processing infinity | 142.27 ms total infinity | embeddings/sec: 56.23 ollama | [GIN] 2024/06/21 - 01:28:21 | 200 | 87.487µs | 127.0.0.1 | HEAD "/" pulling manifest ⠴ time=2024-06-21T01:28:33.263Z level=ERROR source=images.go:978 msg="jwt token does not contain 3 parts" infinity | INFO 2024-06-21 01:28:34,209 infinity_emb INFO: Getting select_model.py:86 infinity | timings for batch_size=8 and avg tokens per infinity | sentence=512 infinity | 4.79 ms tokenization infinity | 7364.73 ms inference infinity | 0.29 ms post-processing infinity | 7369.80 ms total infinity | embeddings/sec: 1.09 infinity | INFO 2024-06-21 01:28:34,212 infinity_emb INFO: model select_model.py:87 infinity | warmed up, between 1.09-56.23 embeddings/sec at infinity | batch_size=8 infinity | INFO 2024-06-21 01:28:34,222 infinity_emb INFO: select_model.py:57 infinity | model=mixedbread-ai/mxbai-rerank-xsmall-v1 infinity | selected, using engine=torch and device=None ollama | [GIN] 2024/06/21 - 01:28:34 | 200 | 12.623310701s | 127.0.0.1 | POST "/api/pull" pulling manifest ollama | pulling 405b56374e02... 100% ▕████████████████▏ 934 MB ollama | pulling 62fbfd9ed093... 100% ▕████████████████▏ 182 B ollama | pulling c156170b718e... 100% ▕████████████████▏ 11 KB ollama | pulling f02dd72bb242... 100% ▕████████████████▏ 59 B ollama | pulling c9f5e9ffbc5f... 100% ▕████████████████▏ 485 B ollama | verifying sha256 digest ollama | writing manifest ollama | removing any unused layers ollama | success infinity | INFO 2024-06-21 01:28:35,051 CrossEncoder.py:110 infinity | sentence_transformers.cross_encoder.CrossEncoder infinity | INFO: Use pytorch device: cpu infinity | INFO 2024-06-21 01:28:35,301 infinity_emb INFO: Adding acceleration.py:30 infinity | optimizations via Huggingface optimum. infinity | WARNING 2024-06-21 01:28:35,303 infinity_emb WARNING: acceleration.py:41 infinity | BetterTransformer is not available for model: <class infinity | 'transformers.models.deberta_v2.modeling_deberta_v2. infinity | DebertaV2ForSequenceClassification'> Continue infinity | without bettertransformer modeling code. infinity | INFO 2024-06-21 01:28:35,726 infinity_emb INFO: Getting select_model.py:80 infinity | timings for batch_size=8 and avg tokens per infinity | sentence=2 infinity | 0.68 ms tokenization infinity | 43.69 ms inference infinity | 0.01 ms post-processing infinity | 44.38 ms total infinity | embeddings/sec: 180.27 infinity | INFO 2024-06-21 01:28:40,044 infinity_emb INFO: Getting select_model.py:86 infinity | timings for batch_size=8 and avg tokens per infinity | sentence=512 infinity | 9.10 ms tokenization infinity | 2110.77 ms inference infinity | 0.04 ms post-processing infinity | 2119.91 ms total infinity | embeddings/sec: 3.77 infinity | INFO 2024-06-21 01:28:40,047 infinity_emb INFO: model select_model.py:87 infinity | warmed up, between 3.77-180.27 embeddings/sec at infinity | batch_size=8 infinity | INFO 2024-06-21 01:28:40,049 infinity_emb INFO: batch_handler.py:321 infinity | creating batching engine infinity | INFO 2024-06-21 01:28:40,051 infinity_emb INFO: ready batch_handler.py:384 infinity | to batch requests. infinity | INFO 2024-06-21 01:28:40,053 infinity_emb INFO: batch_handler.py:321 infinity | creating batching engine infinity | INFO 2024-06-21 01:28:40,055 infinity_emb INFO: ready batch_handler.py:384 infinity | to batch requests. infinity | INFO 2024-06-21 01:28:40,057 infinity_emb INFO: infinity_server.py:59 infinity | infinity | ♾️ Infinity - Embedding Inference Server infinity | MIT License; Copyright (c) 2023 Michael Feil infinity | Version 0.0.42 infinity | infinity | Open the Docs via Swagger UI: infinity | http://0.0.0.0:7997/docs infinity | infinity | Access model via 'GET': infinity | curl http://0.0.0.0:7997/models infinity | infinity | INFO: Application startup complete. infinity | INFO: Uvicorn running on http://0.0.0.0:7997 (Press CTRL+C to quit) ollama | [GIN] 2024/06/21 - 01:28:41 | 200 | 51.703µs | 127.0.0.1 | HEAD "/" ollama | [GIN] 2024/06/21 - 01:28:41 | 200 | 422.044µs | 127.0.0.1 | GET "/api/tags" infinity | INFO: 127.0.0.1:57644 - "GET /health HTTP/1.1" 200 OK cognita-backend | Installing Prisma CLI cognita-backend | * Install prebuilt node (22.3.0) ..... done. cognita-backend | Prisma schema loaded from backend/database/schema.prisma cognita-backend | Datasource "db": PostgreSQL database "cognita-config", schema "public" at "host.docker.internal:5432" cognita-backend | cognita-backend | Error: P1001: Can't reach database server at host.docker.internal:5432 cognita-backend | cognita-backend | Please make sure your database server is running at host.docker.internal:5432. Gracefully stopping... (press Ctrl+C again to force) dependency failed to start: container cognita-backend is unhealthy (cognita) Ubuntu@0068-kci-prxmx10127:~/cognita$ `

[S1LV3RJ1NX]

The error says the version of docker-compose / docker is obsolete. Can you check what version you are using? Also, maybe for linux systems you might have to change the DOCKER_HOST env variable. I'll check if I can add you to an internal slack and help you debug the issue. Until then please check the above versions and let me know.

I am on Mac system and following are my versions, the file builds successfully: Docker Compose version v2.24.6-desktop.1 Docker version 25.0.3, build 4debf41

[S1LV3RJ1NX]

@fahdmirza , can I get your email? I'll add you to the internal cognita-channel

[fahdmirza]

Sure, its fahd.mirza.c@gmail.com .

[chiragjn]

I am closing this, but please feel free to re-open if things don't work out for you on lastest main :)