search

qdrant / fastembed

Fast, Accurate, Lightweight Python library to make State of the Art Embedding
https://qdrant.github.io/fastembed/
Apache License 2.0
1.4k stars 100 forks source link

Promote Huggingface Hub to first class citizen #81

Closed NirantK closed 8 months ago

NirantK commented 9 months ago

The latest plans are in the most recent comment at the end

Error Handling improvements will come from two main improvements:

  1. Migrating away from GCP to Huggingface Hub completely
    1. This will reduce the edge cases we need to maintain, including file renaming and similar code too
  2. For models which we push to HF Hub, we can add a “name” and “sources” field —
    1. where the name is what HF Hub base model and sources is a list of community or Qdrant models

This issue is about the first one.

How to push models?

This is a good reference contribution: https://huggingface.co/weakit-v/bge-base-en-v1.5-onnx/tree/main

This is what we should aim to replicate as much as we can. We'll have these models under the Qdrant Huggingface Hub account instead. So they'd be something like: qdrant/bge-base-en-v1.5-onnx

{
   name: "BAAI/bge-base-en-v1.5",
   sources: ["qdrant/bge-base-en-v1.5-onnx", "weakit-v/bge-base-en-v1.5-onnx"]
}

We'll have to do this for each model, one at a time:

In this process, we deprecate the following models by not porting them from GCP to HF Hub on our account:

  1. BAAI/bge-small-en
  2. BAAI/bge-small-zh-v1.5
  3. BAAI/bge-base-en
generall commented 8 months ago

As a first step, I would implement support for both - HF and arbitrary links like it is right now. Then if we will see benefit of complete migration, we can continue

NirantK commented 8 months ago

Motivation

Why should we consider upgrading Huggingface Hub as a download option from JinaEmbedding specific to all Embedding models?

  1. That makes it easier to support new models — including community additions
  2. Improves error handling e.g. we can add multiple sources for the same model, consistent naming convention
  3. Download stats for models built by Qdrant via Huggingface Hub

Where will we change things?

We'll change things in the Embedding class. We'll add a download_from_hf or similar function. We will continue to support existing models via GCS (arbitrary URLs via requests) in addition to Huggingface Hub.

Download Function will:

  1. First, check if there is a Huggingface model or source
  2. If yes, download from there and corresponding loaders will be used
  3. If not, we check from Google Cloud Storage/URL

At the end of this issue, users will be able to:

How to push models?

This is a good reference contribution: https://huggingface.co/weakit-v/bge-base-en-v1.5-onnx/tree/main

This is what we should aim to replicate as much as we can. We'll have these models under the Qdrant Huggingface Hub account instead. So they'd be something like: qdrant/bge-base-en-v1.5-onnx

{
   name: "BAAI/bge-base-en-v1.5",
   sources: ["qdrant/bge-base-en-v1.5-onnx", "weakit-v/bge-base-en-v1.5-onnx"]
}

We'll have to do this for each model, one at a time:

NirantK commented 8 months ago

cc @Anush008 Added the 4 models here: https://huggingface.co/Qdrant. This should unblock you completely

When a -Q and without Q suffix both are available, prefer the one without the Q suffix.

The other pattern is that of Jina: https://huggingface.co/jinaai/jina-embeddings-v2-small-en/tree/main

I'd recommend that we find a way to handle this without downloading PyTorch files at all. If we can't find that, open a new issue and I'll coordinate with the Jina folks.

x4080 commented 7 months ago

is "Qdrant/bge-m3-onnx" already supported ?