Error: Server does not support event stream content type, it returned application/json

[scchess]

I want to deploy a few open source models with the chat UI. I started a simple model with:

model=tiiuae/falcon-7b-instruct
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:1.3 --model-id $model

And then simply added this new endpoint to my chat UI deployment. While my deployment can connect to the model server, I'm getting an error:

Error: Server does not support event stream content type, it returned application/json
    at streamingRequest (file:///home/scchess/hugging-face-tgi/chat-ui/node_modules/@huggingface/inference/dist/index.mjs:292:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Proxy.textGenerationStream (file:///home/scchess/hugging-face-tgi/chat-ui/node_modules/@huggingface/inference/dist/index.mjs:673:3)
    at async Module.generateFromDefaultEndpoint (/home/scchess/hugging-face-tgi/chat-ui/src/lib/server/generateFromDefaultEndpoint.ts:11:20)
    at async Module.summarize (/home/scchess/hugging-face-tgi/chat-ui/src/lib/server/summarize.ts:28:10)
    at async eval (/home/scchess/hugging-face-tgi/chat-ui/src/routes/conversation/[id]/+server.ts:167:26)

How should I disable streaming for the chat UI?

[rohitm]

Same problem here! It seems that the chat ui client doesn't support "block responses" and the hugging face inference endpoints don't support "streaming responses" by default (at least on their handler.py template). There is some vague chatter about creating a custom endpoint handler to be able to stream responses but I'm not finding how to do that. This is a big gap in my opinion.