"This message exceeded the context limit for this LLM. Please select a different LLM and try again, or shorten your prompt." issue

[hra42]

Software

Desktop Application

Operating System / Platform

macOS

Your Pieces OS Version

10.1.15

Early Access Program

• [ ] Yes, this is related to an Early Access Program feature.

Kindly describe the bug and include as much detail as possible on what you were doing so we can reproduce the bug.

I wanted to demo the power of pieces using the day one challenge of advent of code.

I received an error that the messages exceeded the context:

I tried in a new chat and is started working as expected:

Logs: pOS: log-12032024.txt PfD: log-12032024.txt

Here is the complete message for reproduction:

Can you give a short plan on how to implement this:
--- Day 1: Historian Hysteria ---

The Chief Historian is always present for the big Christmas sleigh launch, but nobody has seen him in months! Last anyone heard, he was visiting locations that are historically significant to the North Pole; a group of Senior Historians has asked you to accompany them as they check the places they think he was most likely to visit.

As each location is checked, they will mark it on their list with a star. They figure the Chief Historian must be in one of the first fifty places they'll look, so in order to save Christmas, you need to help them get fifty stars on their list before Santa takes off on December 25th.

Collect stars by solving puzzles. Two puzzles will be made available on each day in the Advent calendar; the second puzzle is unlocked when you complete the first. Each puzzle grants one star. Good luck!

You haven't even left yet and the group of Elvish Senior Historians has already hit a problem: their list of locations to check is currently empty. Eventually, someone decides that the best place to check first would be the Chief Historian's office.

Upon pouring into the office, everyone confirms that the Chief Historian is indeed nowhere to be found. Instead, the Elves discover an assortment of notes and lists of historically significant locations! This seems to be the planning the Chief Historian was doing before he left. Perhaps these notes can be used to determine which locations to search?

Throughout the Chief's office, the historically significant locations are listed not by name but by a unique number called the location ID. To make sure they don't miss anything, The Historians split into two groups, each searching the office and trying to create their own complete list of location IDs.

There's just one problem: by holding the two lists up side by side (your puzzle input), it quickly becomes clear that the lists aren't very similar. Maybe you can help The Historians reconcile their lists?

For example:

3   4
4   3
2   5
1   3
3   9
3   3
Maybe the lists are only off by a small amount! To find out, pair up the numbers and measure how far apart they are. Pair up the smallest number in the left list with the smallest number in the right list, then the second-smallest left number with the second-smallest right number, and so on.

Within each pair, figure out how far apart the two numbers are; you'll need to add up all of those distances. For example, if you pair up a 3 from the left list with a 7 from the right list, the distance apart is 4; if you pair up a 9 with a 3, the distance apart is 6.

In the example list above, the pairs and distances would be as follows:

The smallest number in the left list is 1, and the smallest number in the right list is 3. The distance between them is 2.
The second-smallest number in the left list is 2, and the second-smallest number in the right list is another 3. The distance between them is 1.
The third-smallest number in both lists is 3, so the distance between them is 0.
The next numbers to pair up are 3 and 4, a distance of 1.
The fifth-smallest numbers in each list are 3 and 5, a distance of 2.
Finally, the largest number in the left list is 4, while the largest number in the right list is 9; these are a distance 5 apart.
To find the total distance between the left list and the right list, add up the distances between all of the pairs you found. In the example above, this is 2 + 1 + 0 + 1 + 2 + 5, a total distance of 11!

Your actual left and right lists contain many location IDs. What is the total distance between your lists?

[mark-at-pieces]

hey Henry @hra42,

Was talking with @sam-at-pieces on this issue and we have both experienced this same issue, however it occurs rather infrequently.

I believe we should also bring @mack-at-pieces into this as well to take a deeper look into the PFD logs.

I was unable to reproduce this issue with the same input, however I will dig a bit deeper in the mean time.

[hra42]

Maybe for your information, I don't have that many pieces saved in that instance, currently only the 2 generated from the startup from #464.

From looking at the logs of PfD I can't figure out what happened, for me at looks like Pieces did exactly what was supposed to happen within the workflow. Maybe you can see more on your end.

Funny is that I received the error almost immediately, I would guess thet pieces does try to add some context to the message, either from onboarding or profile or something with vector db doesn't work properly so we receive more context then we should what makes the request fail. Hard to troubleshoot.

[sam-at-pieces]

Thanks for raising this @hra42. The strange thing is that their should be no additional context at all added to this message and looking at what is being passed to the LLM it is ~1000 tokens so should easily fit in the context window. It's a tough one! I will take a deeper look tomorrow am. Would you be able to share what LLM you were using when you saw these? Was it Claude by any chance?

[hra42]

@sam-at-pieces Yes, I used Claude 3.5 Sonnet.

[sam-at-pieces]

Great so looking into this and chatting with @mack-at-pieces, we throw this error when the stream completes and the output is empty on the pfd side.

I've confirmed that they empty outputs are returned from the vertex AOT For Claude + Gemini when would expect an error code (I've sent too many tokens, incorrect parameters etc.). I suspect that the error is being suppressed somewhere in the AOT and an empty prediction is being returned. The following issues can cause this: 1) When harmful content is detected (notorious number of false positives here) 2) When there is an internal error on the google side 3) When services are temporarily unavailable 4) when incorrect parameters are passed 5) When the message content is too long

I've confirmed that in production we will never pass a message that is too long to the LLM so this response is likely a result of one of the first four cases.

@mack-at-pieces I would recommend that we change the error message to reflect this to avoid confusion. We should prompt the user to switch models or retry (recommended in the vertex AI docs). I'm thinking something like this.

@mark-at-pieces are we repressing some of these errors on the AOT side? We should be getting 500s back for exceeding the prompt length but I always get a completed request, just with an empty prediction. If we surface these I can automate retrying.

[hra42]

@sam-at-pieces outstanding work! Thank you so much for diving so deep into this error!

If you need a contact person on GCP, please let me know, I might be able to help with this.