huggingface / chat-ui

Open source codebase powering the HuggingChat app
https://huggingface.co/chat
Apache License 2.0
7.17k stars 1.03k forks source link

Extend Cloudflare endpoint to support non-Hugging Face models #1364

Closed wgcv closed 1 month ago

wgcv commented 1 month ago

I'm experiencing an issue with the Cloudflare endpoint, which seems to only work with Hugging Face models. I'd like to propose a change to make the endpoint more flexible and support other models, such as Cloudflare's models (e.g., @cf/mistral/mistral-7b-instruct-v0.1).

Current code:

MODELS=`[
    {
  "name" : "meta-llama/meta-llama-3-8b-instruct",
  "parameters": {
    "stop": ["<|im_end|>"]
  },
  "endpoints" : [
    {
      "type" : "cloudflare",
    }
  ]
}
]`

https://github.com/huggingface/chat-ui/blob/7692f71091e027a99195d6bcff6a2ec8c9180983/src/lib/server/endpoints/cloudflare/endpointCloudflare.ts#L19

Proposed change:

MODELS=`[
    {
  "name" : "@hf/nexusflow/starling-lm-7b-beta",
  "parameters": {
    "stop": ["<|im_end|>"]
  },
  "endpoints" : [
    {
      "type" : "cloudflare",
    }
  ]
}
]`
const apiURL = `https://api.cloudflare.com/client/v4/accounts/${accountId}/ai/run/${model.id}`; 

By making this change, we can enable the use of a wider range of models with the Cloudflare endpoint, including Cloudflare's own models.

nsarrazin commented 1 month ago

I think this makes sense for sure, maybe we could check if the string startswith @ and if not we add the @hf/ prefix to prevent breaking changes with existing deployments?