Open Jerry-VW opened 3 months ago
Same here
Hey, I don't know if you've already solved the problem, but I managed to fix it, and maybe it can help you.
My issue was that I was using message.content
in a map.
messages.map((m, i) => (
<ChatMessage
content={m.content}
/>
))
This was causing multiple updates.
I solved the problem by passing messages
directly to the final component:
<ChatMessage content={messages} />
I hope this helps you.
I also experience this issue. I only have it using the route hadler. After changing to the new rsc/ai i have not seen it. It must have something to do with the re-renders when streaming.
Sure! Here’s the translation to English:
Hey, I don't know if you've already solved the problem, but I managed to fix it, and maybe it can help you.
My issue was that I was using
message.content
in a map.messages.map((m, i) => ( <ChatMessage content={m.content} /> ))
This was causing multiple updates.
I solved the problem by passing
messages
directly to the final component:<ChatMessage content={messages} />
I hope this helps you.
So after reading this message here i think i finely solved the issue... Spend so much time on it xD
So before i had the chat message displayed like this here:
const ChatMessage: FC<ChatMessageProps> = ({ messages }) => {
const [isCopied, setIsCopied] = useState(false);
const router = useRouter();
const componentsAI: Partial<Components> = {
a: ({ href, children }) => (
<a
href={href}
onClick={(e) => {
e.preventDefault();
if (href) {
router.push(href);
}
}}
>
{children}
</a>
),
code({ className, children, ...props }) {
const match = /language-(\w+)/.exec(className || '');
const language = match && match[1] ? match[1] : '';
const inline = !language;
if (inline) {
return (
<code className={className} {...props}>
{children}
</code>
);
}
return (
<div
style={{
position: 'relative',
borderRadius: '5px',
padding: '20px',
marginTop: '20px',
maxWidth: '100%'
}}
>
<span
style={{
position: 'absolute',
top: '0',
left: '5px',
fontSize: '0.8em',
textTransform: 'uppercase'
}}
>
{language}
</span>
<div
style={{
overflowX: 'auto',
maxWidth: '1100px'
}}
>
<pre style={{ margin: '0' }}>
<code className={className} {...props}>
{children}
</code>
</pre>
</div>
</div>
);
}
};
const componentsUser: Partial<Components> = {
a: ({ href, children }) => (
<a
href={href}
onClick={(e) => {
e.preventDefault();
if (href) {
router.push(href);
}
}}
>
{children}
</a>
)
};
const copyToClipboard = (str: string): void => {
void window.navigator.clipboard.writeText(str);
};
const handleCopy = (content: string) => {
copyToClipboard(content);
setIsCopied(true);
setTimeout(() => setIsCopied(false), 1000);
};
return (
<>
{messages.map((m, index) => (
<ListItem
key={`${m.id}-${index}`}
sx={
m.role === 'user'
? messageStyles.userMessage
: messageStyles.aiMessage
}
>
<Box
sx={{
position: 'absolute',
top: '10px',
left: '10px'
}}
>
{m.role === 'user' ? (
<PersonIcon sx={{ color: '#4caf50' }} />
) : (
<AndroidIcon sx={{ color: '#607d8b' }} />
)}
</Box>
{m.role === 'assistant' && (
<Box
sx={{
position: 'absolute',
top: '5px',
right: '5px',
cursor: 'pointer',
display: 'flex',
alignItems: 'center',
justifyContent: 'center',
width: 24,
height: 24
}}
onClick={() => handleCopy(m.content)}
>
{isCopied ? (
<CheckCircleIcon fontSize="inherit" />
) : (
<ContentCopyIcon fontSize="inherit" />
)}
</Box>
)}
<Box sx={{ overflowWrap: 'break-word' }}>
<Typography
variant="caption"
sx={{ fontWeight: 'bold', display: 'block' }}
>
{m.role === 'user' ? 'You' : 'AI'}
</Typography>
{m.role === 'user' ? (
<ReactMarkdown
components={componentsUser}
remarkPlugins={[remarkGfm, remarkMath]}
rehypePlugins={[rehypeHighlight]}
>
{m.content}
</ReactMarkdown>
) : (
<ReactMarkdown
components={componentsAI}
remarkPlugins={[remarkGfm, remarkMath]}
rehypePlugins={[[rehypeHighlight, highlightOptionsAI]]}
>
{m.content}
</ReactMarkdown>
)}
</Box>
</ListItem>
))}
</>
);
};
and then in the output return
<List
sx={{
marginBottom: '120px'
}}
>
<ChatMessage messages={messages} />
</List>
This caused exhaustions of the max states depth.
However, after changing the the structure to:
const MemoizedMessage = memo(({ message }: { message: Message }) => {
const [isCopied, setIsCopied] = useState(false);
const router = useRouter();
const componentsAI: Partial<Components> = {
a: ({ href, children }) => (
<a
href={href}
onClick={(e) => {
e.preventDefault();
if (href) {
router.push(href);
}
}}
>
{children}
</a>
),
code({ className, children, ...props }) {
const match = /language-(\w+)/.exec(className || '');
const language = match && match[1] ? match[1] : '';
const inline = !language;
if (inline) {
return (
<code className={className} {...props}>
{children}
</code>
);
}
return (
<div
style={{
position: 'relative',
borderRadius: '5px',
padding: '20px',
marginTop: '20px',
maxWidth: '100%'
}}
>
<span
style={{
position: 'absolute',
top: '0',
left: '5px',
fontSize: '0.8em',
textTransform: 'uppercase'
}}
>
{language}
</span>
<div
style={{
overflowX: 'auto',
maxWidth: '1100px'
}}
>
<pre style={{ margin: '0' }}>
<code className={className} {...props}>
{children}
</code>
</pre>
</div>
</div>
);
}
};
const componentsUser: Partial<Components> = {
a: ({ href, children }) => (
<a
href={href}
onClick={(e) => {
e.preventDefault();
if (href) {
router.push(href);
}
}}
>
{children}
</a>
)
};
const copyToClipboard = (str: string): void => {
void window.navigator.clipboard.writeText(str);
};
const handleCopy = (content: string) => {
copyToClipboard(content);
setIsCopied(true);
setTimeout(() => setIsCopied(false), 1000);
};
return (
<ListItem
sx={
message.role === 'user'
? messageStyles.userMessage
: messageStyles.aiMessage
}
>
<Box
sx={{
position: 'absolute',
top: '10px',
left: '10px'
}}
>
{message.role === 'user' ? (
<PersonIcon sx={{ color: '#4caf50' }} />
) : (
<AndroidIcon sx={{ color: '#607d8b' }} />
)}
</Box>
{message.role === 'assistant' && (
<Box
sx={{
position: 'absolute',
top: '5px',
right: '5px',
cursor: 'pointer',
display: 'flex',
alignItems: 'center',
justifyContent: 'center',
width: 24,
height: 24
}}
onClick={() => handleCopy(message.content)}
>
{isCopied ? (
<CheckCircleIcon fontSize="inherit" />
) : (
<ContentCopyIcon fontSize="inherit" />
)}
</Box>
)}
<Box sx={{ overflowWrap: 'break-word' }}>
<Typography
variant="caption"
sx={{ fontWeight: 'bold', display: 'block' }}
>
{message.role === 'user' ? 'You' : 'AI'}
</Typography>
{message.role === 'user' ? (
<ReactMarkdown
components={componentsUser}
remarkPlugins={[remarkGfm, remarkMath]}
rehypePlugins={[rehypeHighlight]}
>
{message.content}
</ReactMarkdown>
) : (
<ReactMarkdown
components={componentsAI}
remarkPlugins={[remarkGfm, remarkMath]}
rehypePlugins={[[rehypeHighlight, highlightOptionsAI]]}
>
{message.content}
</ReactMarkdown>
)}
</Box>
</ListItem>
);
});
MemoizedMessage.displayName = 'MemoizedMessage';
const ChatMessage: FC<ChatMessageProps> = ({ messages }) => {
return (
<>
{messages.map((message, index) => (
<MemoizedMessage key={`${message.id}-${index}`} message={message} />
))}
</>
);
};
the lag issues and the maximum update depth exceeded seems to have disappeared completely. In Dev i was only able to ever get many 4 or 5 messages before chrome just gave up, now i can get pretty much as many as i want!
I tried to reproduce the bug with useCompletion
and gpt-4o
(4096 completion tokens). However, it did not show up for me.
@Jerry-VW Can you provide me with code to reproduce? Ideally some modification of the next/useCompletion example (which I tried w gpt-4o):
I have this older branch of my example project where i have the exact same problem using the "useChat" and an api route.
https://github.com/ElectricCodeGuy/SupabaseAuthWithSSR/tree/0643777a348d63b7d0b0260ad99e5054be1b4062
In dev environment it would always lag out and crash my browser after a few msg. In production i have not experienced the same level of lag. Here it is on par with other chatbots i have tried. So after maybe 20 msg the UI can begin to lock up. I'm 90% convinced it have something to do with the .map function and re-renders, but i have not successfully found the root cause.
I discovered that when using the streaming method, errors occur when the response message reaches a certain length. Even with streaming, no errors occur if the response message is short. I suspect that this issue arises because the component updates every time a streaming input comes in.
I'm looking for a minimal examples because it's unclear to me whether this is an issue with useChat / useCompletion or with the other React code.
@ElectricCodeGuy your examples has a lot of other code, which makes it hard to pinpoint the issue
@Jerry-VW is this for a single response / completion or for a long chat?
@choipd i tried to produce a very long message (max tokens) with no issues. however, i have a pretty fast machine and that might also play a role here
I ran into this trying to render chat.messages[0].content
as a prop. Fix it with a slice/map:
chat.messages.slice(0,1).map( message => <Component content={message.content} />
I think the problem is React picking up on the .content
as a deep dependency to watch for updates. Running it through a map keep the reactivity on the messages
maybe.
Any update?
TBH i think a lot of the previous answers are incorrect. I think it comes from long responses which cause react to rerender too many times in a row (50 is the limit) which depends entirely on the response size. Something that has temporarily fixed the issue for me has been to create a queue which chunks the stream values into length n values before joining and streaming it to the frontend. This increases the maximum response size by n times and can be adjusted as needed.
Here is a simplified example from my serverAction:
...
const streamableValue = createStreamableValue("");
const streamChunks = async () => {
let queue = [];
for await (const chunk of stream) {
const content = chunk.choices[0]?.delta.content;
if (content == null) {
continue;
}
queue.push(content);
if (queue.length >= 8) {
streamableValue.append(queue.join(""));
queue = [];
}
}
streamableValue.append(queue.join(""));
streamableValue.done();
};
streamChunks();
return { value: streamableValue.value };
I haven't hit the error since this but it doesn't mean it couldn't occur with a really large response. I assume the real solution would be to somehow 'give react a break' while streaming.
@arnab710 if you want an update kindly prepare a minimum reproduction of the issue for the maintainers.
This code mentions "synchronously", I wonder if instead of setting state each render we should use startTransition as that doesn't block the UI when updating.
// Count the number of times the root synchronously re-renders without // finishing. If there are too many, it indicates an infinite update loop.
A state update marked as a Transition will be interrupted by other state updates. For example, if you update a chart component inside a Transition, but then start typing into an input while the chart is in the middle of a re-render, React will restart the rendering work on the chart component after handling the input state update.
For me, The markdown library was causing performance issues due to the rapid influx of data chunks, which it was unable to process in real-time (i.e. backpressure
).
I resolved the lag effectively by optimizing my markdown component.
@lgrammel
Having the same issue when messages is a dependency for something else. Tried wrapping it in a debounce but it loses real time streaming.
I thinkuseChat()
(and similar streaming hooks) should have a built-in support for a prop like debounceInterval
which does all the debouncing behind the scenes.
Here's where it's coming from in case it helps. It's happening on every single component using the SDK to stream content.
I believe this needs to be throttled inside of the SDK directly.
@oalexdoda throttling would impact the stream consumption significantly, since we are using backpressure and the reading speed depends on the client speed. Before I move to a fix, I'd like to see a minimal reproduction that I can run myself. You mentioned that it's related to a message dependency. Would you mind putting together a minimal reproduction, either as PR or as a repo, so I can investigate?
So i fixed this using use-debounce
and the defaults i set seem to be the lowest numbers here to not get the error and still feel like its streaming
import { useCompletion as useCompletionAI } from 'ai/react';
import { useDebounce } from 'use-debounce';
type UseCompletionArgs = Parameters<typeof useCompletionAI>[0] & {
delay?: number;
maxWait?: number;
};
export const useCompletion = ({ delay = 250, maxWait = 250, ...args }: UseCompletionArgs) => {
const { completion, ...rest } = useCompletionAI(args);
const [debouncedCompletion] = useDebounce(completion, delay, { maxWait });
return {
...rest,
completion: debouncedCompletion
};
};
To summarize:
Considerations:
wait
must be configurable with a reasonable default (e.g. 50ms)
Description
Use useCompletion from AI SDK to call gpt-4o that will have a long response in streaming mode. It will hang the UI. Looks like its updating completion's state in a very fast pace.
Code example
No response
Additional context
No response