"Invalid State: Controller is already closed" error when trying to use chat-ui with trained llama2 model on the HF platform

[ardaakdere]

When I try to run the llama2 model card that I trained, with chat-ui in Space using Nvidia A10G small, I get no response.

I see the following error in the container log:

""" 07:43:00 3|index | TypeError [ERR_INVALID_STATE]: Invalid state: Controller is already closed 07:43:00 3|index | at new NodeError (node:internal/errors:399:5) 07:43:00 3|index | at ReadableStreamDefaultController.enqueue (node:internal/webstreams/readablestream:1036:13) 07:43:00 3|index | at update (file:///app/build/server/chunks/_server.ts-2f619497.js:451:20) 07:43:00 3|index | at file:///app/build/server/chunks/_server.ts-2f619497.js:461:13 07:43:00 3|index | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 07:43:00 3|index | at async Object.start (file:///app/build/server/chunks/_server.ts-2f619497.js:552:7) { 07:43:00 3|index | code: 'ERR_INVALID_STATE' 07:43:00 3|index | } """

Chat UI files:

env.local.template file used:

# Use .env.local to change these variables
# DO NOT EDIT THIS FILE WITH SENSITIVE DATA

MONGODB_URL=${MONGODB_URL}
MONGODB_DB_NAME=chat-ui
MONGODB_DIRECT_CONNECTION=false

COOKIE_NAME=chat-ui
HF_ACCESS_TOKEN=#hf_<token> from from https://huggingface.co/settings/token

# used to activate search with web functionality. disabled if not defined
SERPAPI_KEY=#your serpapi key here

# Parameters to enable "Sign in with HF"
OPENID_CLIENT_ID=
OPENID_CLIENT_SECRET=
OPENID_SCOPES="openid profile" # Add "email" for some providers like Google that do not provide preferred_username
OPENID_PROVIDER_URL=https://huggingface.co # for Google, use https://accounts.google.com

# 'name', 'userMessageToken', 'assistantMessageToken' are required
MODELS=`[
  {
    "name": "${MODEL_NAME}",
    "chatPromptTemplate": "${MODEL_PROMPT_TEMPLATE}",
    "preprompt": "",
    "promptExamples": [
      {
        "title": "Python Fibonacci",
        "prompt": "How can I write a Python function to generate the nth Fibonacci number?"
      }, {
        "title": "What is a meme?",
        "prompt": "What is a meme, and what's the history behind this word?"
      }, {
        "title": "Regex",
        "prompt": "Create a regex to extract dates from logs"
      }
    ],
    "endpoints": [
      {
        "type": "tgi",
        "url": "http://127.0.0.1:8080"
      }
    ],
    "parameters": ${MODEL_PARAMS}
  }
]`
OLD_MODELS=`[]`# any removed models, `{ name: string, displayName?: string, id?: string }`

PUBLIC_ORIGIN=${SPACE_HOST}
PUBLIC_SHARE_PREFIX=${SPACE_HOST}/r
PUBLIC_GOOGLE_ANALYTICS_ID=#G-XXXXXXXX / Leave empty to disable
PUBLIC_DEPRECATED_GOOGLE_ANALYTICS_ID=#UA-XXXXXXXX-X / Leave empty to disable
PUBLIC_ANNOUNCEMENT_BANNERS=`[
  {
    "title": "Chat UI is now open sourced on GitHub",
    "linkTitle": "GitHub repo",
    "linkHref": "https://github.com/huggingface/chat-ui"
  }
]`

PARQUET_EXPORT_DATASET=
PARQUET_EXPORT_HF_TOKEN=
PARQUET_EXPORT_SECRET=

PUBLIC_APP_NAME=${APP_NAME} # name used as title throughout the app
PUBLIC_APP_ASSETS=chatui # used to find logos & favicons in static/$PUBLIC_APP_ASSETS
PUBLIC_APP_COLOR=${APP_COLOR} # can be any of tailwind colors: https://tailwindcss.com/docs/customizing-colors#default-color-palette
PUBLIC_APP_DATA_SHARING=#set to 1 to enable disclaimers & options about data sharing
PUBLIC_APP_DATA_DISCLAIMER=#set to 1 to enable disclaimers about model outputs

entrypoint.sh.template used:

#!/bin/bash

# Start the local Mongo database
mongod &

# Start the text-generation-inference process
text-generation-launcher --model-id ${MODEL_NAME} --num-shard 1 --port 8080 --trust-remote-code &

# Wait for text-generation-inference to start
curl --retry 60 --retry-delay 10 --retry-connrefused http://127.0.0.1:8080/health

# Start the chat-ui process
pm2 start /app/build/index.js -i $CPU_CORES --no-daemon &

# Wait for any process to exit
wait -n

# Exit with status of process that exited first
exit $?

The interesting thing is that when I send a one word text, although I can't get a reply from the chat section in the interface, I can see the short reply from the preview in the left menu. I am sharing a screenshot below explaining this situation:

another example for the same situation: (run on Nvidia 10G Large)

Screenshot of the container error I mentioned at the beginning of the Issue:

chat-ui uses "text-generation-launcher" to run the model. People who previously solved the same error for llama.cpp solved the problem by setting the "np" argument to 2. When I looked at the documentation of "text-generation-launcher" I couldn't see a similar argument. I did tests by playing with the VALIDATION_WORKERS, NUM_SHARD arguments here ( https://huggingface.co/docs/text-generation-inference/basic_tutorials/launcher). But I couldn't get any result.

Could there be something I might have missed here? Does anyone have a solution you can suggest on this?

Thanks.

[mikelfried]

Hi,

Try modify the https://github.com/huggingface/chat-ui/blob/main/src/routes/conversation/%5Bid%5D/%2Bserver.ts file,

moving the await summarizeIfNeeded; at line 325 a few lines before.

change from

await collections.conversations.updateOne(
  {
      _id: convId,
  },
  {
      $set: {
          messages,
          title: conv?.title,
          updatedAt: new Date(),
      },
  }
);

update({
  type: "finalAnswer",
  text: messages[messages.length - 1].content,
});

await summarizeIfNeeded;

to

await summarizeIfNeeded;

await collections.conversations.updateOne(
  {
      _id: convId,
  },
  {
      $set: {
          messages,
          title: conv?.title,
          updatedAt: new Date(),
      },
  }
);

update({
  type: "finalAnswer",
  text: messages[messages.length - 1].content,
});

please respond if it is working for you now.

[ardaakdere]

Thanks! @mikelfried

I have tried your solution. Now it works in a few cases but not in all.

After the solution you suggested, I no longer get the log error mentioned in the issue, that's really good news.

Here I can verify that it responds to a longer text from the menu on the left, but in the chat input, I get the feedback "Sorry, something went wrong. Please try again.":

When I look at the logs in the same scenario:

Here, you can check the screen recording: https://github.com/huggingface/chat-ui/assets/54773283/f4128023-f76e-4df5-b0d3-c5cee9bc054d

A console error that I have only encountered once (and then never again, may not be directly related to this topic, i'm not sure):

I didn't get a visible error for "Sorry, something went wrong. Please try again."

Do you have any idea about this?

[loganlebanoff]

Here's what fixed it for me https://github.com/huggingface/chat-ui/issues/1169#issuecomment-2173309506