rossanodr
commented
6 months ago Same here
Closed Jerry-VW closed 1 month ago
rossanodr
commented
6 months ago Same here
rossanodr
commented
6 months ago Hey, I don't know if you've already solved the problem, but I managed to fix it, and maybe it can help you.
My issue was that I was using message.content in a map.
messages.map((m, i) => (
<ChatMessage
content={m.content}
/>
))This was causing multiple updates.
I solved the problem by passing messages directly to the final component:
<ChatMessage content={messages} />I hope this helps you.
ElectricCodeGuy
commented
6 months ago I also experience this issue. I only have it using the route hadler. After changing to the new rsc/ai i have not seen it. It must have something to do with the re-renders when streaming.
Sure! Here’s the translation to English:
Hey, I don't know if you've already solved the problem, but I managed to fix it, and maybe it can help you.
My issue was that I was using
message.contentin a map.messages.map((m, i) => ( <ChatMessage content={m.content} /> ))This was causing multiple updates.
I solved the problem by passing
messagesdirectly to the final component:<ChatMessage content={messages} />I hope this helps you.
So after reading this message here i think i finely solved the issue... Spend so much time on it xD
So before i had the chat message displayed like this here:
const ChatMessage: FC<ChatMessageProps> = ({ messages }) => {
const [isCopied, setIsCopied] = useState(false);
const router = useRouter();
const componentsAI: Partial<Components> = {
a: ({ href, children }) => (
<a
href={href}
onClick={(e) => {
e.preventDefault();
if (href) {
router.push(href);
}
}}
>
{children}
</a>
),
code({ className, children, ...props }) {
const match = /language-(\w+)/.exec(className || '');
const language = match && match[1] ? match[1] : '';
const inline = !language;
if (inline) {
return (
<code className={className} {...props}>
{children}
</code>
);
}
return (
<div
style={{
position: 'relative',
borderRadius: '5px',
padding: '20px',
marginTop: '20px',
maxWidth: '100%'
}}
>
<span
style={{
position: 'absolute',
top: '0',
left: '5px',
fontSize: '0.8em',
textTransform: 'uppercase'
}}
>
{language}
</span>
<div
style={{
overflowX: 'auto',
maxWidth: '1100px'
}}
>
<pre style={{ margin: '0' }}>
<code className={className} {...props}>
{children}
</code>
</pre>
</div>
</div>
);
}
};
const componentsUser: Partial<Components> = {
a: ({ href, children }) => (
<a
href={href}
onClick={(e) => {
e.preventDefault();
if (href) {
router.push(href);
}
}}
>
{children}
</a>
)
};
const copyToClipboard = (str: string): void => {
void window.navigator.clipboard.writeText(str);
};
const handleCopy = (content: string) => {
copyToClipboard(content);
setIsCopied(true);
setTimeout(() => setIsCopied(false), 1000);
};
return (
<>
{messages.map((m, index) => (
<ListItem
key={`${m.id}-${index}`}
sx={
m.role === 'user'
? messageStyles.userMessage
: messageStyles.aiMessage
}
>
<Box
sx={{
position: 'absolute',
top: '10px',
left: '10px'
}}
>
{m.role === 'user' ? (
<PersonIcon sx={{ color: '#4caf50' }} />
) : (
<AndroidIcon sx={{ color: '#607d8b' }} />
)}
</Box>
{m.role === 'assistant' && (
<Box
sx={{
position: 'absolute',
top: '5px',
right: '5px',
cursor: 'pointer',
display: 'flex',
alignItems: 'center',
justifyContent: 'center',
width: 24,
height: 24
}}
onClick={() => handleCopy(m.content)}
>
{isCopied ? (
<CheckCircleIcon fontSize="inherit" />
) : (
<ContentCopyIcon fontSize="inherit" />
)}
</Box>
)}
<Box sx={{ overflowWrap: 'break-word' }}>
<Typography
variant="caption"
sx={{ fontWeight: 'bold', display: 'block' }}
>
{m.role === 'user' ? 'You' : 'AI'}
</Typography>
{m.role === 'user' ? (
<ReactMarkdown
components={componentsUser}
remarkPlugins={[remarkGfm, remarkMath]}
rehypePlugins={[rehypeHighlight]}
>
{m.content}
</ReactMarkdown>
) : (
<ReactMarkdown
components={componentsAI}
remarkPlugins={[remarkGfm, remarkMath]}
rehypePlugins={[[rehypeHighlight, highlightOptionsAI]]}
>
{m.content}
</ReactMarkdown>
)}
</Box>
</ListItem>
))}
</>
);
};and then in the output return
<List
sx={{
marginBottom: '120px'
}}
>
<ChatMessage messages={messages} />
</List>This caused exhaustions of the max states depth.
However, after changing the the structure to:
const MemoizedMessage = memo(({ message }: { message: Message }) => {
const [isCopied, setIsCopied] = useState(false);
const router = useRouter();
const componentsAI: Partial<Components> = {
a: ({ href, children }) => (
<a
href={href}
onClick={(e) => {
e.preventDefault();
if (href) {
router.push(href);
}
}}
>
{children}
</a>
),
code({ className, children, ...props }) {
const match = /language-(\w+)/.exec(className || '');
const language = match && match[1] ? match[1] : '';
const inline = !language;
if (inline) {
return (
<code className={className} {...props}>
{children}
</code>
);
}
return (
<div
style={{
position: 'relative',
borderRadius: '5px',
padding: '20px',
marginTop: '20px',
maxWidth: '100%'
}}
>
<span
style={{
position: 'absolute',
top: '0',
left: '5px',
fontSize: '0.8em',
textTransform: 'uppercase'
}}
>
{language}
</span>
<div
style={{
overflowX: 'auto',
maxWidth: '1100px'
}}
>
<pre style={{ margin: '0' }}>
<code className={className} {...props}>
{children}
</code>
</pre>
</div>
</div>
);
}
};
const componentsUser: Partial<Components> = {
a: ({ href, children }) => (
<a
href={href}
onClick={(e) => {
e.preventDefault();
if (href) {
router.push(href);
}
}}
>
{children}
</a>
)
};
const copyToClipboard = (str: string): void => {
void window.navigator.clipboard.writeText(str);
};
const handleCopy = (content: string) => {
copyToClipboard(content);
setIsCopied(true);
setTimeout(() => setIsCopied(false), 1000);
};
return (
<ListItem
sx={
message.role === 'user'
? messageStyles.userMessage
: messageStyles.aiMessage
}
>
<Box
sx={{
position: 'absolute',
top: '10px',
left: '10px'
}}
>
{message.role === 'user' ? (
<PersonIcon sx={{ color: '#4caf50' }} />
) : (
<AndroidIcon sx={{ color: '#607d8b' }} />
)}
</Box>
{message.role === 'assistant' && (
<Box
sx={{
position: 'absolute',
top: '5px',
right: '5px',
cursor: 'pointer',
display: 'flex',
alignItems: 'center',
justifyContent: 'center',
width: 24,
height: 24
}}
onClick={() => handleCopy(message.content)}
>
{isCopied ? (
<CheckCircleIcon fontSize="inherit" />
) : (
<ContentCopyIcon fontSize="inherit" />
)}
</Box>
)}
<Box sx={{ overflowWrap: 'break-word' }}>
<Typography
variant="caption"
sx={{ fontWeight: 'bold', display: 'block' }}
>
{message.role === 'user' ? 'You' : 'AI'}
</Typography>
{message.role === 'user' ? (
<ReactMarkdown
components={componentsUser}
remarkPlugins={[remarkGfm, remarkMath]}
rehypePlugins={[rehypeHighlight]}
>
{message.content}
</ReactMarkdown>
) : (
<ReactMarkdown
components={componentsAI}
remarkPlugins={[remarkGfm, remarkMath]}
rehypePlugins={[[rehypeHighlight, highlightOptionsAI]]}
>
{message.content}
</ReactMarkdown>
)}
</Box>
</ListItem>
);
});
MemoizedMessage.displayName = 'MemoizedMessage';
const ChatMessage: FC<ChatMessageProps> = ({ messages }) => {
return (
<>
{messages.map((message, index) => (
<MemoizedMessage key={`${message.id}-${index}`} message={message} />
))}
</>
);
};the lag issues and the maximum update depth exceeded seems to have disappeared completely. In Dev i was only able to ever get many 4 or 5 messages before chrome just gave up, now i can get pretty much as many as i want!
lgrammel
commented
6 months ago I tried to reproduce the bug with useCompletion and gpt-4o (4096 completion tokens). However, it did not show up for me.
@Jerry-VW Can you provide me with code to reproduce? Ideally some modification of the next/useCompletion example (which I tried w gpt-4o):
ElectricCodeGuy
commented
6 months ago I have this older branch of my example project where i have the exact same problem using the "useChat" and an api route.
https://github.com/ElectricCodeGuy/SupabaseAuthWithSSR/tree/0643777a348d63b7d0b0260ad99e5054be1b4062
In dev environment it would always lag out and crash my browser after a few msg. In production i have not experienced the same level of lag. Here it is on par with other chatbots i have tried. So after maybe 20 msg the UI can begin to lock up. I'm 90% convinced it have something to do with the .map function and re-renders, but i have not successfully found the root cause.
choipd
commented
6 months ago I discovered that when using the streaming method, errors occur when the response message reaches a certain length. Even with streaming, no errors occur if the response message is short. I suspect that this issue arises because the component updates every time a streaming input comes in.
lgrammel
commented
6 months ago I'm looking for a minimal examples because it's unclear to me whether this is an issue with useChat / useCompletion or with the other React code.
@ElectricCodeGuy your examples has a lot of other code, which makes it hard to pinpoint the issue
@Jerry-VW is this for a single response / completion or for a long chat?
@choipd i tried to produce a very long message (max tokens) with no issues. however, i have a pretty fast machine and that might also play a role here
bioshazard
commented
5 months ago I ran into this trying to render chat.messages[0].content as a prop. Fix it with a slice/map:
chat.messages.slice(0,1).map( message => <Component content={message.content} />I think the problem is React picking up on the .content as a deep dependency to watch for updates. Running it through a map keep the reactivity on the messages maybe.
arnab710
commented
5 months ago Any update?
ted-marozzi
commented
5 months ago TBH i think a lot of the previous answers are incorrect. I think it comes from long responses which cause react to rerender too many times in a row (50 is the limit) which depends entirely on the response size. Something that has temporarily fixed the issue for me has been to create a queue which chunks the stream values into length n values before joining and streaming it to the frontend. This increases the maximum response size by n times and can be adjusted as needed.
Here is a simplified example from my serverAction:
...
const streamableValue = createStreamableValue("");
const streamChunks = async () => {
let queue = [];
for await (const chunk of stream) {
const content = chunk.choices[0]?.delta.content;
if (content == null) {
continue;
}
queue.push(content);
if (queue.length >= 8) {
streamableValue.append(queue.join(""));
queue = [];
}
}
streamableValue.append(queue.join(""));
streamableValue.done();
};
streamChunks();
return { value: streamableValue.value };I haven't hit the error since this but it doesn't mean it couldn't occur with a really large response. I assume the real solution would be to somehow 'give react a break' while streaming.
ted-marozzi
commented
5 months ago @arnab710 if you want an update kindly prepare a minimum reproduction of the issue for the maintainers.
ted-marozzi
commented
5 months ago This code mentions "synchronously", I wonder if instead of setting state each render we should use startTransition as that doesn't block the UI when updating.
// Count the number of times the root synchronously re-renders without // finishing. If there are too many, it indicates an infinite update loop.
A state update marked as a Transition will be interrupted by other state updates. For example, if you update a chart component inside a Transition, but then start typing into an input while the chart is in the middle of a re-render, React will restart the rendering work on the chart component after handling the input state update.
arnab710
commented
5 months ago For me, The markdown library was causing performance issues due to the rapid influx of data chunks, which it was unable to process in real-time (i.e. backpressure).
I resolved the lag effectively by optimizing my markdown component.
oalexdoda
commented
5 months ago @lgrammel
Having the same issue when messages is a dependency for something else. Tried wrapping it in a debounce but it loses real time streaming.
I thinkuseChat() (and similar streaming hooks) should have a built-in support for a prop like debounceInterval which does all the debouncing behind the scenes.
oalexdoda
commented
5 months ago Here's where it's coming from in case it helps. It's happening on every single component using the SDK to stream content.
oalexdoda
commented
5 months ago I believe this needs to be throttled inside of the SDK directly.
lgrammel
commented
4 months ago @oalexdoda throttling would impact the stream consumption significantly, since we are using backpressure and the reading speed depends on the client speed. Before I move to a fix, I'd like to see a minimal reproduction that I can run myself. You mentioned that it's related to a message dependency. Would you mind putting together a minimal reproduction, either as PR or as a repo, so I can investigate?
Pulseline-Tech
commented
4 months ago So i fixed this using use-debounce and the defaults i set seem to be the lowest numbers here to not get the error and still feel like its streaming
import { useCompletion as useCompletionAI } from 'ai/react';
import { useDebounce } from 'use-debounce';
type UseCompletionArgs = Parameters<typeof useCompletionAI>[0] & {
delay?: number;
maxWait?: number;
};
export const useCompletion = ({ delay = 250, maxWait = 250, ...args }: UseCompletionArgs) => {
const { completion, ...rest } = useCompletionAI(args);
const [debouncedCompletion] = useDebounce(completion, delay, { maxWait });
return {
...rest,
completion: debouncedCompletion
};
};To summarize:
Considerations:
wait must be configurable with a reasonable default (e.g. 50ms)
arkaydeus
commented
2 months ago Still getting this error in the latest version. Is there a working manual workaround at the moment?
HirotoShioi
commented
2 months ago @arkaydeus
I had the same issue with my project and this is how I've solved it (with react):
The workaround seems to be to wrap the Message rendering component with memo
This is the code I found on postgres.new
import { memo } from 'react'
export type ChatMessageProps = {
message: Message
isLast: boolean
}
function ChatMessage({ message, isLast }: ChatMessageProps) {
// implementation of rendering Message
}
// Memoizing is important here - otherwise React continually
// re-renders previous messages unnecessarily (big performance hit)
export default memo(ChatMessage, (prevProps, nextProps) => {
// Always re-render the last message to fix a bug where `useChat()`
// doesn't trigger a re-render when multiple tool calls are added
// to the same message. Otherwise shallow compare.
return (
!nextProps.isLast &&
prevProps.isLast === nextProps.isLast &&
prevProps.message === nextProps.message
)
})How the ChatMessage is being used at chat.tsx
{messages.map((message, i) => (
<ChatMessage
key={message.id}
message={message}
isLast={i === messages.length - 1}
/>
))}I'm currently using a custom fork of useChat and useCompletion that I need to update every time the ai package gets updated. It's the only way I could come up with to prevent the maximum stack depth (even after trying memo). Basically. using throttle from lodash to prevent the update callbacks from firing too often.
useChat
useCompletion
It's not sustainable but it works for now. The other observation I have here is that if the component where you use useChat throws any sort of error (i.e. some hydration error, or anything else), whether in itself or its children, the maximum stack depth will still be reached (probably because of the console.error throwing to the browser).
Or if you console.log anything at all outside of a useEffect, again - it throws the maximum stack depth.
Not sure if this is a React compiler issue (I'm on Next 14 still), or if a permanent fix can be baked into the SDK. Most of the time it errors on very long context conversations, and I've seen it even crash on users in production (to the application error white screen of death).
jhangmez
commented
1 month ago @arkaydeus
I had the same issue with my project and this is how I've solved it (with react):
The workaround seems to be to wrap the Message rendering component with memo
This is the code I found on postgres.new
import { memo } from 'react' export type ChatMessageProps = { message: Message isLast: boolean } function ChatMessage({ message, isLast }: ChatMessageProps) { // implementation of rendering Message } // Memoizing is important here - otherwise React continually // re-renders previous messages unnecessarily (big performance hit) export default memo(ChatMessage, (prevProps, nextProps) => { // Always re-render the last message to fix a bug where `useChat()` // doesn't trigger a re-render when multiple tool calls are added // to the same message. Otherwise shallow compare. return ( !nextProps.isLast && prevProps.isLast === nextProps.isLast && prevProps.message === nextProps.message ) })How the
ChatMessageis being used at chat.tsx{messages.map((message, i) => ( <ChatMessage key={message.id} message={message} isLast={i === messages.length - 1} /> ))}
I've tried this solution but it doesn't work, I will be waiting for the solution that provides the last reply
lgrammel
commented
1 month ago lgrammel
commented
1 month ago @ai-sdk/react@0.0.70 has a experimental_throttle setting for useChat and useCompletion. You can set it to a ms amount to specify the throttling interval.
Description
Use useCompletion from AI SDK to call gpt-4o that will have a long response in streaming mode. It will hang the UI. Looks like its updating completion's state in a very fast pace.
Code example
No response
Additional context
No response