`stream` is not supported for this model

[newUserForTesting]

Hello Eperts, Trying to run https://github.com/huggingface/chat-ui by providing models like EleutherAI/pythia-1b, gpt2-large. With all these models, there is this consitent error {"error":["Error in stream: stream is not supported for this model"]} Although I can see that hosted inference API for these models are working well from their hugging face pages like this: https://huggingface.co/gpt2-large Could someone please help?

[nsarrazin]

Hey! I can try to help.

First, i think those models are not actually streaming, they just look like that in the front-end but if you look at the network request they are not actually streaming tokens.

But I think chat-ui should still work with non streaming models. If you show me your .env (while hiding any api keys) maybe I can help you figure out what's going on ?

[aniket-synth]

I am trying to use https://github.com/huggingface/chat-ui with https://github.com/vllm-project/vllm/tree/main openai endpoint. Eventhough the request goes to the vllm server, I constantly get the error in the UI: Server does not support event stream content type, it returned text/event-stream; charset=utf-8. Could someone please help

[julien-c]

maybe @Narsil or @OlivierDehaene would know?

[dillfrescott]

I need help with this. Ive tried everything and no luck. :/

[dillfrescott]

SyntaxError: Unexpected token 'D', " [DONE]" is not valid JSON
    at JSON.parse (<anonymous>)
    at parseGeneratedText (/root/chat-ui/src/routes/conversation/[id]/+server.ts:242:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async saveMessage (/root/chat-ui/src/routes/conversation/[id]/+server.ts:155:26)

Its saying this as well. Even though this is exactly whats supposed to happen from an openai api endpoint

[newUserForTesting]

@nsarrazin Here is .env file (hopefully you weren't asking for .env.local)

# Use .env.local to change these variables
# DO NOT EDIT THIS FILE WITH SENSITIVE DATA

MONGODB_URL=#your mongodb URL here
MONGODB_DB_NAME=chat-ui
MONGODB_DIRECT_CONNECTION=false

COOKIE_NAME=hf-chat
HF_ACCESS_TOKEN=#hf_<token> from from https://huggingface.co/settings/token
HF_API_ROOT=https://api-inference.huggingface.co/models

# used to activate search with web functionality. disabled if none are defined. choose one of the following:
SERPER_API_KEY=#your serper.dev api key here
SERPAPI_KEY=#your serpapi key here

# Parameters to enable "Sign in with HF"
OPENID_CLIENT_ID=
OPENID_CLIENT_SECRET=
OPENID_SCOPES="openid profile" # Add "email" for some providers like Google that do not provide preferred_username
OPENID_PROVIDER_URL=https://huggingface.co # for Google, use https://accounts.google.com

# Parameters to enable a global mTLS context for client fetch requests
USE_CLIENT_CERTIFICATE=false
CERT_PATH=#
KEY_PATH=#
CA_PATH=#
CLIENT_KEY_PASSWORD=#
REJECT_UNAUTHORIZED=true

# 'name', 'userMessageToken', 'assistantMessageToken' are required
MODELS=`[
  {
    "name": "OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5",
    "datasetName": "OpenAssistant/oasst1",
    "description": "A good alternative to ChatGPT",
    "websiteUrl": "https://open-assistant.io",
    "userMessageToken": "<|prompter|>",
    "assistantMessageToken": "<|assistant|>",
    "messageEndToken": "</s>",
    "preprompt": "Below are a series of dialogues between various people and an AI assistant. The AI tries to be helpful, polite, honest, sophisticated, emotionally aware, and humble-but-knowledgeable. The assistant is happy to help with almost anything, and will do its best to understand exactly what is needed. It also tries to avoid giving false or misleading information, and it caveats when it isn't entirely sure about the right answer. That said, the assistant is practical and really does its best, and doesn't let caution get too much in the way of being useful.\n-----\n",
    "promptExamples": [
      {
        "title": "Write an email from bullet list",
        "prompt": "As a restaurant owner, write a professional email to the supplier to get these products every week: \n\n- Wine (x10)\n- Eggs (x24)\n- Bread (x12)"
      }, {
        "title": "Code a snake game",
        "prompt": "Code a basic snake game in python, give explanations for each step."
      }, {
        "title": "Assist in a task",
        "prompt": "How do I make a delicious lemon cheesecake?"
      }
    ],
    "parameters": {
      "temperature": 0.9,
      "top_p": 0.95,
      "repetition_penalty": 1.2,
      "top_k": 50,
      "truncate": 1000,
      "max_new_tokens": 1024
    }
  }
]`
OLD_MODELS=`[]`# any removed models, `{ name: string, displayName?: string, id?: string }`

PUBLIC_ORIGIN=#https://huggingface.co
PUBLIC_SHARE_PREFIX=#https://hf.co/chat
PUBLIC_GOOGLE_ANALYTICS_ID=#G-XXXXXXXX / Leave empty to disable
PUBLIC_DEPRECATED_GOOGLE_ANALYTICS_ID=#UA-XXXXXXXX-X / Leave empty to disable
PUBLIC_ANNOUNCEMENT_BANNERS=`[
    {
    "title": "New Models!",
    "linkTitle": "Announcement",
    "linkHref": "https://huggingface.co/blog/llama2"
  }
]`

PARQUET_EXPORT_DATASET=
PARQUET_EXPORT_HF_TOKEN=
PARQUET_EXPORT_SECRET=

RATE_LIMIT= # requests per minute
MESSAGES_BEFORE_LOGIN=# how many messages a user can send in a conversation before having to login. set to 0 to force login right away

PUBLIC_APP_NAME=ChatUI # name used as title throughout the app
PUBLIC_APP_ASSETS=chatui # used to find logos & favicons in static/$PUBLIC_APP_ASSETS
PUBLIC_APP_COLOR=blue # can be any of tailwind colors: https://tailwindcss.com/docs/customizing-colors#default-color-palette
PUBLIC_APP_DATA_SHARING=#set to 1 to enable options & text regarding data sharing
PUBLIC_APP_DISCLAIMER=#set to 1 to show a disclaimer on login page

# PUBLIC_APP_NAME=HuggingChat
# PUBLIC_APP_ASSETS=huggingchat
# PUBLIC_APP_COLOR=yellow
# PUBLIC_APP_DATA_SHARING=1
# PUBLIC_APP_DISCLAIMER=1

[Narsil]

I am trying to use https://github.com/huggingface/chat-ui with https://github.com/vllm-project/vllm/tree/main openai endpoint. Eventhough the request goes to the vllm server, I constantly get the error in the UI: Server does not support event stream content type, it returned text/event-stream; charset=utf-8. Could someone please help

I'm not sure chat-ui and vllm are made to work with each other at the moment. For chat-ui we use https://github.com/huggingface/text-generation-inference as a backend, and there are most likely differences in the JSON keys and formats they expect.

Should be simple glue, but I haven' t really looked at the http server implementation.

[julien-c]

@newUserForTesting i edited your post for better formatting

[Vancelott]

At the moment, the only working model for textGenerationStream that I've found is "google/flan-t5-xxl". All of the other models that I've attempted to use resulted in getting the above-mentioned error "Error in stream: stream is not supported for this model".

[scchess]

I have the same error with tiiuae/falcon-7b-instruct. How do I switch streaming off?