search

chroma-core / chroma

the AI-native open-source embedding database
https://www.trychroma.com/
Apache License 2.0
14.97k stars 1.26k forks source link

[Feature Request]: Add functionality for dimension size parameter in embedding_functions.OpenAIEmbeddingFunction() as introduced by OPENAI for new embedding models #2576

Closed Raghavgarg12 closed 3 days ago

Raghavgarg12 commented 2 months ago

Describe the problem

It would be helpful if we get add functionality for dimension size parameter in embedding_functions.OpenAIEmbeddingFunction() as introduced by OPENAI for new embedding models

Describe the proposed solution

Just add the parameter to be sent during request to openai's server

Alternatives considered

No response

Importance

would make my life easier

Additional Information

No response

tazarov commented 2 months ago

@Raghavgarg12, thanks for raising this. We have a fairly long-standing PR that addresses this - https://github.com/chroma-core/chroma/pull/1977. Let me know if this addresses your issue.

Raghavgarg12 commented 2 months ago

openai_ef = embedding_functions.OpenAIEmbeddingFunction( api_key=OPENAI_API_KEY, model_name="text-embedding-3-small", dimensions=512 )

openai_ef = embedding_functions.OpenAIEmbeddingFunction(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

TypeError: OpenAIEmbeddingFunction.init() got an unexpected keyword argument 'dimensions'

tazarov commented 2 months ago

@Raghavgarg12, that is expected as the PR is not yet merged :). Let me rebase it and you can check it out and test it.

Raghavgarg12 commented 2 months ago

Right. Thankyou.

tazarov commented 2 months ago

Let me re-open and we'll close once we get the PR merged.

tazarov commented 2 months ago

@Raghavgarg12, rebased. You can test it by installing Chroma from the PR branch:

IMPORTANT: if you are using persistent client version < 0.5.5 running the below will migrate your existing database so you'll have to use 0.5.5 going forward. I'd suggest running your tests on a fresh new DB to verify the PR.

pip install git+https://github.com/chroma-core/chroma.git@trayan-04-05-feat_adding_dimensions_to_openai_ef