Open solvek opened 9 months ago
With the info I see here, my first assumption is that this port configuration issue.
chromadb.HttpClient
to port='443'
, I assume on your Cloud Run container URL.Uvicorn running on http://0.0.0.0:8000
According to this cloud run container article, cloud run forwards incoming requests to port 8080
within the container, which doesn't match the port 8000
that chroma listens on
Niftily-enough you might be able to fix this with a single gcloud command.
gcloud run services update SERVICE --port PORT
, where SERVICE
is the name of your running service and PORT
is what you want gcloud to forward requests to (Chroma's 8000
)
I'm curious if that does the trick, please let me know!
Tried this command
gcloud run services update SERVICE --port PORT
, whereSERVICE
is the name of your running service andPORT
is what you want gcloud to forward requests to (Chroma's8000
)
But this didn't help. The port 8000 was already configured.
According to my logs from server it seems the requests are forwarded fine to port 8000 and are handled by server... howerver the path /api/v1/tenants/default_tenant
seems to be not working :(
@tazarov any ideas here?
It seems I managed to sort this out.
I am going to document how to deploy chroma server as a Google Cloud Run Service and how to connect to it from Google Cloud Run Job.
However before this want to resolve some other issues I have.
The first issue is I cannot reset the db even thougt I have set allow_reset=True
Creating client
resetting_client = chromadb.HttpClient(
host=CHROMA_SERVER_HOST,
port=CHROMA_SERVER_HTTP_PORT,
ssl = True,
settings=Settings(anonymized_telemetry=False, allow_reset=True)
)
Then
resetting_client.reset()
However getting error
Exception: {"error":"ValueError('Resetting is not allowed by this configuration (to enable it, set
allow_resetto
Truein your Settings() or include
ALLOW_RESET=TRUEin your environment variables)')"}
@solvek, reset is a flag you need to set on the server-side e.g. your Cloud Run Service cofig needs to contain ALLOW_RESET=TRUE
env var.
ALLOW_RESET=TRUE
on server side helped me. Thanks.
Another strange issue is I am getting in logs warnings about deprecated configuration (and telemetry btw):
2023-11-28 17:47:46.927 EET
Failed to send telemetry event ClientStartEvent: [91mYou are using a deprecated configuration of Chroma.
2023-11-28 17:47:46.927 EET
Connected to chromadb
2023-11-28 17:47:46.927 EET
[94mIf you do not have data you wish to migrate, you only need to change how you construct
2023-11-28 17:47:46.927 EET
your Chroma client. Please see the "New Clients" section of https://docs.trychroma.com/migration.
2023-11-28 17:47:46.927 EET
________________________________________________________________________________________________
2023-11-28 17:47:46.927 EET
If you do have data you wish to migrate, we have a migration tool you can use in order to
2023-11-28 17:47:46.927 EET
migrate your data to the new Chroma architecture.
2023-11-28 17:47:46.927 EET
Please `pip install chroma-migrate` and run `chroma-migrate` to migrate your data and then
2023-11-28 17:47:46.927 EET
change how you construct your Chroma client.
2023-11-28 17:47:46.927 EET
See https://docs.trychroma.com/migration for more information or join our discord at https://discord.gg/8g5FESbj for help![0m
This is my code on the client side:
print(f"Starting Task #{TASK_INDEX}, Attempt #{TASK_ATTEMPT}...")
client = chromadb.HttpClient(
host=CHROMA_SERVER_HOST,
port=CHROMA_SERVER_HTTP_PORT,
ssl = True,
settings=Settings(anonymized_telemetry=False)
)
print("Connected to chromadb")
default_ef = embedding_functions.DefaultEmbeddingFunction()
collection = client.get_or_create_collection(
"all-my-documents",
embedding_function=default_ef)
collection.add(
documents=["This is document1", "This is document2"],
metadatas=[{"source": "notion"}, {"source": "google-docs"}], # filter on these!
ids=["doc1", "doc2"], # unique for each doc
)
print("Chroma docs added")
And the 3rd issue I have an error:
Task #0, Attempt #0 failed: You must provide an embedding function to compute embeddings.https://docs.trychroma.com/embeddings
However I am providing embedding function as you can see in my code snippet above.
@solvek, what Chroma version is you client?
EDIT: The client code looks fine. However, you should note that embedding_functions.DefaultEmbeddingFunction()
will trigger the download of onyx runtime + all-MiniLM model (83MB file). If you run this in a serverless, it might only be optimal if there is some caching. As an alternative, I suggest you switch to either OpenAI, Cohere or HF inference API.
Chroma client version is 0.4.18 Even thoght I am setting embedding function I am getting error when I add data to my collection. (On production I will use Palm2 however for now I want to have any simple working solution without errors)
Task #0, Attempt #0 failed: You must provide an embedding function to compute embeddings.https://docs.trychroma.com/embeddings
P.S. Probably some dependencies are missing. I am using chromadb-client
package on the Cloud Job side.
Additional information: It seems I am having all these issues because I am user chromadb-client
package instead of chromadb
.
chromadb
does not work on the cloud job at all. Described this in another issue here: https://github.com/GoogleCloudPlatform/buildpacks/issues/366
@solvek, chroma-client is the client-only version of Chroma, and it has very little dependencies:
dependencies = [
'requests >= 2.28',
'pydantic>=1.9',
'numpy >= 1.22.5',
'posthog >= 2.4.0',
'typing_extensions >= 4.5.0',
'overrides >= 7.3.1',
]
As far as I know GCR operates on top of Knative functions. Knative functions usually require you to build an image. Do you have the Dockerfile of your image?
Nvm I found it from the linked issue:
"telethon",
"langchain >= 0.0.338",
"google-cloud-aiplatform",
"chromadb ==0.4.18"
]
Can you change chromadb==0.4.18
to chromadb-client==0.4.18.dev0
I do not have a Dockerfile. I am building the container from python code.
This is my requirements.txt
:
opentelemetry-sdk
opentelemetry-exporter-otlp
chromadb-client
I tried also to run the client in Google Colab and really if I install package chromadb
then I do not have the issue with embeddings, when I install only chromadb-client
I am having this error there as well.
So it seems some dependency is missing.
With chromadb-client==0.4.18.dev0
still getting that error with embeddings
By the way when using chromadb-client
I had to add also
opentelemetry-sdk
opentelemetry-exporter-otlp
Otherwise it would not work even if I have anonymized_telemetry=False
Alright then you might be onto something. Let me do some digging on my side for a moment.
@solvek, Looking at the code, this is intended behaviour:
def DefaultEmbeddingFunction() -> Optional[EmbeddingFunction[Documents]]:
if is_thin_client:
return None
else:
return ONNXMiniLM_L6_V2()
is_thin_client
is set for the chromadb-client package. The issue you're facing is that to run the ONNX runtime and all, you need a few more dependencies, so we've opted not to include those for the lightweight client.
As a workaround, I would suggest trying the following:
onnxruntime>=1.14.1
to your requirementsef = embedding_functions.DefaultEmbeddingFunction()
try ef = embedding_functions.ONNXMiniLM_L6_V2()
(this still might fail due to some deps)@solvek, to summarize our discussion in Discord.
chromadb-client
package where missing OTEL deps were causing failuretokenizers
package needed installef.DOWNLOAD_PATH = "models/ONNXMiniLM_L6_V2"
needed to be added to resolve the permissions errorThe bottom line is while the above worked, it is not a workable solution as GCR is a serverless function for which downloading 80MB on each run is a non-starter. The appropriate solution is to use an external API EF (OpenAI, Cohere, Vertex, HF etc).
I think with the above learnings and the documentation about how to set and run Chroma thin client in a GCR (for which I'll create a separate issue), we can safely close this issue.
Wouldn't you want to mount a persistent volume to this as well? I've only found an in-memory volume mount option for Cloud Run and the data won't survive. Am I missing something?
@florind, granted you could store the data in the GCR persistent volume and check if the model exists in it, if yes then load and use that.
Generally speaking doing this in a serverless fashion is not optimal as loading and running models is not the fastest thing. I would much rather use a separate service either an API or another Container service in Google with HF's embedding server. We recently added support and instructions how to run it - https://docs.trychroma.com/embeddings/hugging-face-embedding-server (in docker)
What happened?
Created a service. Deployed directly docker from Docker Hub.
Now trying connect via HttpClient:
Versions
PYTHON_VERSION=3.10.13 chromadb 0.4.18
Relevant log output
Client size
Server side