rossanodr
commented
3 months ago Same here
Open Jerry-VW opened 3 months ago
rossanodr
commented
3 months ago Same here
rossanodr
commented
3 months ago Hey, I don't know if you've already solved the problem, but I managed to fix it, and maybe it can help you.
My issue was that I was using message.content in a map.
messages.map((m, i) => (
<ChatMessage
content={m.content}
/>
))This was causing multiple updates.
I solved the problem by passing messages directly to the final component:
<ChatMessage content={messages} />I hope this helps you.
ElectricCodeGuy
commented
3 months ago I also experience this issue. I only have it using the route hadler. After changing to the new rsc/ai i have not seen it. It must have something to do with the re-renders when streaming.
Sure! Here’s the translation to English:
Hey, I don't know if you've already solved the problem, but I managed to fix it, and maybe it can help you.
My issue was that I was using
message.contentin a map.messages.map((m, i) => ( <ChatMessage content={m.content} /> ))This was causing multiple updates.
I solved the problem by passing
messagesdirectly to the final component:<ChatMessage content={messages} />I hope this helps you.
So after reading this message here i think i finely solved the issue... Spend so much time on it xD
So before i had the chat message displayed like this here:
const ChatMessage: FC<ChatMessageProps> = ({ messages }) => {
const [isCopied, setIsCopied] = useState(false);
const router = useRouter();
const componentsAI: Partial<Components> = {
a: ({ href, children }) => (
<a
href={href}
onClick={(e) => {
e.preventDefault();
if (href) {
router.push(href);
}
}}
>
{children}
</a>
),
code({ className, children, ...props }) {
const match = /language-(\w+)/.exec(className || '');
const language = match && match[1] ? match[1] : '';
const inline = !language;
if (inline) {
return (
<code className={className} {...props}>
{children}
</code>
);
}
return (
<div
style={{
position: 'relative',
borderRadius: '5px',
padding: '20px',
marginTop: '20px',
maxWidth: '100%'
}}
>
<span
style={{
position: 'absolute',
top: '0',
left: '5px',
fontSize: '0.8em',
textTransform: 'uppercase'
}}
>
{language}
</span>
<div
style={{
overflowX: 'auto',
maxWidth: '1100px'
}}
>
<pre style={{ margin: '0' }}>
<code className={className} {...props}>
{children}
</code>
</pre>
</div>
</div>
);
}
};
const componentsUser: Partial<Components> = {
a: ({ href, children }) => (
<a
href={href}
onClick={(e) => {
e.preventDefault();
if (href) {
router.push(href);
}
}}
>
{children}
</a>
)
};
const copyToClipboard = (str: string): void => {
void window.navigator.clipboard.writeText(str);
};
const handleCopy = (content: string) => {
copyToClipboard(content);
setIsCopied(true);
setTimeout(() => setIsCopied(false), 1000);
};
return (
<>
{messages.map((m, index) => (
<ListItem
key={`${m.id}-${index}`}
sx={
m.role === 'user'
? messageStyles.userMessage
: messageStyles.aiMessage
}
>
<Box
sx={{
position: 'absolute',
top: '10px',
left: '10px'
}}
>
{m.role === 'user' ? (
<PersonIcon sx={{ color: '#4caf50' }} />
) : (
<AndroidIcon sx={{ color: '#607d8b' }} />
)}
</Box>
{m.role === 'assistant' && (
<Box
sx={{
position: 'absolute',
top: '5px',
right: '5px',
cursor: 'pointer',
display: 'flex',
alignItems: 'center',
justifyContent: 'center',
width: 24,
height: 24
}}
onClick={() => handleCopy(m.content)}
>
{isCopied ? (
<CheckCircleIcon fontSize="inherit" />
) : (
<ContentCopyIcon fontSize="inherit" />
)}
</Box>
)}
<Box sx={{ overflowWrap: 'break-word' }}>
<Typography
variant="caption"
sx={{ fontWeight: 'bold', display: 'block' }}
>
{m.role === 'user' ? 'You' : 'AI'}
</Typography>
{m.role === 'user' ? (
<ReactMarkdown
components={componentsUser}
remarkPlugins={[remarkGfm, remarkMath]}
rehypePlugins={[rehypeHighlight]}
>
{m.content}
</ReactMarkdown>
) : (
<ReactMarkdown
components={componentsAI}
remarkPlugins={[remarkGfm, remarkMath]}
rehypePlugins={[[rehypeHighlight, highlightOptionsAI]]}
>
{m.content}
</ReactMarkdown>
)}
</Box>
</ListItem>
))}
</>
);
};and then in the output return
<List
sx={{
marginBottom: '120px'
}}
>
<ChatMessage messages={messages} />
</List>This caused exhaustions of the max states depth.
However, after changing the the structure to:
const MemoizedMessage = memo(({ message }: { message: Message }) => {
const [isCopied, setIsCopied] = useState(false);
const router = useRouter();
const componentsAI: Partial<Components> = {
a: ({ href, children }) => (
<a
href={href}
onClick={(e) => {
e.preventDefault();
if (href) {
router.push(href);
}
}}
>
{children}
</a>
),
code({ className, children, ...props }) {
const match = /language-(\w+)/.exec(className || '');
const language = match && match[1] ? match[1] : '';
const inline = !language;
if (inline) {
return (
<code className={className} {...props}>
{children}
</code>
);
}
return (
<div
style={{
position: 'relative',
borderRadius: '5px',
padding: '20px',
marginTop: '20px',
maxWidth: '100%'
}}
>
<span
style={{
position: 'absolute',
top: '0',
left: '5px',
fontSize: '0.8em',
textTransform: 'uppercase'
}}
>
{language}
</span>
<div
style={{
overflowX: 'auto',
maxWidth: '1100px'
}}
>
<pre style={{ margin: '0' }}>
<code className={className} {...props}>
{children}
</code>
</pre>
</div>
</div>
);
}
};
const componentsUser: Partial<Components> = {
a: ({ href, children }) => (
<a
href={href}
onClick={(e) => {
e.preventDefault();
if (href) {
router.push(href);
}
}}
>
{children}
</a>
)
};
const copyToClipboard = (str: string): void => {
void window.navigator.clipboard.writeText(str);
};
const handleCopy = (content: string) => {
copyToClipboard(content);
setIsCopied(true);
setTimeout(() => setIsCopied(false), 1000);
};
return (
<ListItem
sx={
message.role === 'user'
? messageStyles.userMessage
: messageStyles.aiMessage
}
>
<Box
sx={{
position: 'absolute',
top: '10px',
left: '10px'
}}
>
{message.role === 'user' ? (
<PersonIcon sx={{ color: '#4caf50' }} />
) : (
<AndroidIcon sx={{ color: '#607d8b' }} />
)}
</Box>
{message.role === 'assistant' && (
<Box
sx={{
position: 'absolute',
top: '5px',
right: '5px',
cursor: 'pointer',
display: 'flex',
alignItems: 'center',
justifyContent: 'center',
width: 24,
height: 24
}}
onClick={() => handleCopy(message.content)}
>
{isCopied ? (
<CheckCircleIcon fontSize="inherit" />
) : (
<ContentCopyIcon fontSize="inherit" />
)}
</Box>
)}
<Box sx={{ overflowWrap: 'break-word' }}>
<Typography
variant="caption"
sx={{ fontWeight: 'bold', display: 'block' }}
>
{message.role === 'user' ? 'You' : 'AI'}
</Typography>
{message.role === 'user' ? (
<ReactMarkdown
components={componentsUser}
remarkPlugins={[remarkGfm, remarkMath]}
rehypePlugins={[rehypeHighlight]}
>
{message.content}
</ReactMarkdown>
) : (
<ReactMarkdown
components={componentsAI}
remarkPlugins={[remarkGfm, remarkMath]}
rehypePlugins={[[rehypeHighlight, highlightOptionsAI]]}
>
{message.content}
</ReactMarkdown>
)}
</Box>
</ListItem>
);
});
MemoizedMessage.displayName = 'MemoizedMessage';
const ChatMessage: FC<ChatMessageProps> = ({ messages }) => {
return (
<>
{messages.map((message, index) => (
<MemoizedMessage key={`${message.id}-${index}`} message={message} />
))}
</>
);
};the lag issues and the maximum update depth exceeded seems to have disappeared completely. In Dev i was only able to ever get many 4 or 5 messages before chrome just gave up, now i can get pretty much as many as i want!
lgrammel
commented
3 months ago I tried to reproduce the bug with useCompletion and gpt-4o (4096 completion tokens). However, it did not show up for me.
@Jerry-VW Can you provide me with code to reproduce? Ideally some modification of the next/useCompletion example (which I tried w gpt-4o):
ElectricCodeGuy
commented
3 months ago I have this older branch of my example project where i have the exact same problem using the "useChat" and an api route.
https://github.com/ElectricCodeGuy/SupabaseAuthWithSSR/tree/0643777a348d63b7d0b0260ad99e5054be1b4062
In dev environment it would always lag out and crash my browser after a few msg. In production i have not experienced the same level of lag. Here it is on par with other chatbots i have tried. So after maybe 20 msg the UI can begin to lock up. I'm 90% convinced it have something to do with the .map function and re-renders, but i have not successfully found the root cause.
choipd
commented
3 months ago I discovered that when using the streaming method, errors occur when the response message reaches a certain length. Even with streaming, no errors occur if the response message is short. I suspect that this issue arises because the component updates every time a streaming input comes in.
lgrammel
commented
3 months ago I'm looking for a minimal examples because it's unclear to me whether this is an issue with useChat / useCompletion or with the other React code.
@ElectricCodeGuy your examples has a lot of other code, which makes it hard to pinpoint the issue
@Jerry-VW is this for a single response / completion or for a long chat?
@choipd i tried to produce a very long message (max tokens) with no issues. however, i have a pretty fast machine and that might also play a role here
bioshazard
commented
2 months ago I ran into this trying to render chat.messages[0].content as a prop. Fix it with a slice/map:
chat.messages.slice(0,1).map( message => <Component content={message.content} />I think the problem is React picking up on the .content as a deep dependency to watch for updates. Running it through a map keep the reactivity on the messages maybe.
arnab710
commented
2 months ago Any update?
ted-marozzi
commented
2 months ago TBH i think a lot of the previous answers are incorrect. I think it comes from long responses which cause react to rerender too many times in a row (50 is the limit) which depends entirely on the response size. Something that has temporarily fixed the issue for me has been to create a queue which chunks the stream values into length n values before joining and streaming it to the frontend. This increases the maximum response size by n times and can be adjusted as needed.
Here is a simplified example from my serverAction:
...
const streamableValue = createStreamableValue("");
const streamChunks = async () => {
let queue = [];
for await (const chunk of stream) {
const content = chunk.choices[0]?.delta.content;
if (content == null) {
continue;
}
queue.push(content);
if (queue.length >= 8) {
streamableValue.append(queue.join(""));
queue = [];
}
}
streamableValue.append(queue.join(""));
streamableValue.done();
};
streamChunks();
return { value: streamableValue.value };I haven't hit the error since this but it doesn't mean it couldn't occur with a really large response. I assume the real solution would be to somehow 'give react a break' while streaming.
ted-marozzi
commented
2 months ago @arnab710 if you want an update kindly prepare a minimum reproduction of the issue for the maintainers.
ted-marozzi
commented
2 months ago This code mentions "synchronously", I wonder if instead of setting state each render we should use startTransition as that doesn't block the UI when updating.
// Count the number of times the root synchronously re-renders without // finishing. If there are too many, it indicates an infinite update loop.
A state update marked as a Transition will be interrupted by other state updates. For example, if you update a chart component inside a Transition, but then start typing into an input while the chart is in the middle of a re-render, React will restart the rendering work on the chart component after handling the input state update.
arnab710
commented
2 months ago For me, The markdown library was causing performance issues due to the rapid influx of data chunks, which it was unable to process in real-time (i.e. backpressure).
I resolved the lag effectively by optimizing my markdown component.
oalexdoda
commented
2 months ago @lgrammel
Having the same issue when messages is a dependency for something else. Tried wrapping it in a debounce but it loses real time streaming.
I thinkuseChat() (and similar streaming hooks) should have a built-in support for a prop like debounceInterval which does all the debouncing behind the scenes.
oalexdoda
commented
2 months ago Here's where it's coming from in case it helps. It's happening on every single component using the SDK to stream content.
oalexdoda
commented
2 months ago I believe this needs to be throttled inside of the SDK directly.
lgrammel
commented
2 months ago @oalexdoda throttling would impact the stream consumption significantly, since we are using backpressure and the reading speed depends on the client speed. Before I move to a fix, I'd like to see a minimal reproduction that I can run myself. You mentioned that it's related to a message dependency. Would you mind putting together a minimal reproduction, either as PR or as a repo, so I can investigate?
Pulseline-Tech
commented
1 month ago So i fixed this using use-debounce and the defaults i set seem to be the lowest numbers here to not get the error and still feel like its streaming
import { useCompletion as useCompletionAI } from 'ai/react';
import { useDebounce } from 'use-debounce';
type UseCompletionArgs = Parameters<typeof useCompletionAI>[0] & {
delay?: number;
maxWait?: number;
};
export const useCompletion = ({ delay = 250, maxWait = 250, ...args }: UseCompletionArgs) => {
const { completion, ...rest } = useCompletionAI(args);
const [debouncedCompletion] = useDebounce(completion, delay, { maxWait });
return {
...rest,
completion: debouncedCompletion
};
};To summarize:
Considerations:
wait must be configurable with a reasonable default (e.g. 50ms)
Description
Use useCompletion from AI SDK to call gpt-4o that will have a long response in streaming mode. It will hang the UI. Looks like its updating completion's state in a very fast pace.
Code example
No response
Additional context
No response