[Bug]: /docker_entrypoint.sh: line 5: 38 Segmentation fault (core dumped) uvicorn chromadb.app:app --workers 1 --host 0.0.0.0 --port 8000 --proxy-headers --log-config log_config.yml

[kevinlu1248]

What happened?

The database seg faulted a few times in deployment. I upgraded the instance recently too.

Versions

Docker image with hash 0e50c7589afc. Docker version 23.0.6, build ef23cbc. Running Vultr with 4vCPU and 8Gb of RAM (should I upgrade further?).

Relevant log output

2023-05-21 20:55:22 WARNING  chromadb.api.models.Collection No embedding_function provided, using default embedding function: SentenceTransformerEmbeddingFunction
2023-05-21 20:55:22 INFO     uvicorn.access  54.172.131.20:43232 - "POST /api/v1/collections HTTP/1.1" 200
/docker_entrypoint.sh: line 5:    39 Segmentation fault      (core dumped) uvicorn chromadb.app:app --workers 1 --host 0.0.0.0 --port 8000 --proxy-headers --log-config log_config.yml
Rebuilding hnsw to ensure architecture compatibility
Collecting hnswlib
  Downloading hnswlib-0.7.0.tar.gz (33 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting numpy
  Downloading numpy-1.24.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 191.9 MB/s eta 0:00:00
Building wheels for collected packages: hnswlib
  Building wheel for hnswlib (pyproject.toml): started
  Building wheel for hnswlib (pyproject.toml): finished with status 'done'
  Created wheel for hnswlib: filename=hnswlib-0.7.0-cp310-cp310-linux_x86_64.whl size=2154605 sha256=9d0baceddc61e13c921928f087d73a27f596f4642d1c44574b24f3258ddf6ed7
  Stored in directory: /tmp/pip-ephem-wheel-cache-7u4g6u8u/wheels/8a/ae/ec/235a682e0041fbaeee389843670581ec6c66872db856dfa9a4
Successfully built hnswlib
Installing collected packages: numpy, hnswlib
  Attempting uninstall: numpy
    Found existing installation: numpy 1.24.3
    Uninstalling numpy-1.24.3:
      Successfully uninstalled numpy-1.24.3
  Attempting uninstall: hnswlib
    Found existing installation: hnswlib 0.7.0
    Uninstalling hnswlib-0.7.0:
      Successfully uninstalled hnswlib-0.7.0
Successfully installed hnswlib-0.7.0 numpy-1.24.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.0.1 -> 23.1.2
[notice] To update, run: pip install --upgrade pip
2023-05-21 20:56:05 INFO     chromadb.telemetry.posthog Anonymized telemetry enabled. See https://docs.trychroma.com/telemetry for more information.
2023-05-21 20:56:05 INFO     chromadb        Running Chroma using direct local API.
2023-05-21 20:56:05 WARNING  chromadb        Using embedded DuckDB without persistence: data will be transient
2023-05-21 20:56:05 INFO     uvicorn.error   Started server process [39]
2023-05-21 20:56:05 INFO     uvicorn.error   Waiting for application startup.
2023-05-21 20:56:05 INFO     uvicorn.error   Application startup complete.
2023-05-21 20:56:05 INFO     uvicorn.error   Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

[kevinlu1248]

Don't mind upgrading the instance since we have lots of free credits but want to make sure that's not the root cause of the problem.

[HammadB]

Is there more log output here? I don't see a segfault trace.

1. How many datapoints do you have
2. what is the dimension of your vectors
3. are you storing documents

You could run out of ram - but want to confirm

[kevinlu1248]

Line 3 has a seg fault but no trace.

1. Each cluster has a few hundred generally
2. We use https://huggingface.co/microsoft/unixcoder-base-nine so the dimension is 768.
3. I believe we are storing documents.

Our code looks something like

    collection.add(
        documents=documents,
        metadatas=metadatas,
        ids=ids,
    )

so I believe it means they are being stored?

[HammadB]

How many "clusters" do you have? I am trying to understand the total data size on one machine.

[kevinlu1248]

I mean collection sorry. I think at the time when it crashed we only had like 5.

[jeffchuber]

Closing this as it is stale. Please let me know if anything else pops up here and we can re-open it.

[kevinlu1248]

@jeffchuber it's not stale, the issue was never really addressed nor fixed.

[jeffchuber]

@kevinlu1248 chroma 0.4 introduced a wholesale rewrite of the backend which I suspect addresses this issue. Do you want to give it a spin?

[Victordeleusse]

Hello,

Still got the same issue when adding already embedded data to my chroma db

Using

def add(ids: OneOrMany[ID],
        embeddings: Optional[OneOrMany[Embedding]] = None,
        metadatas: Optional[OneOrMany[Metadata]] = None,
        documents: Optional[OneOrMany[Document]] = None) -> None

from https://docs.trychroma.com/reference/py-collection