feedback: awful

[MichalKurzewski]

Version

1.8.3

Areas for Improvement

• [ ] UI/UX
• [ ] Onboarding
• [ ] Docs
• [ ] Chat
• [ ] Commands
• [ ] Context
• [X] Response Quality
• [ ] Other

What needs to be improved? Please describe how this affects the user experience.

no idea what needs to be improved, but using Claude3 opus gave me such terrible hallucinated answers i wouldn't even expect from chat gpt3.5. i provided few files as a context, it seems to understand the files, but gave totally idiotic answers twice.

Describe the solution you'd like to see

any solution that would at least be possible. the question i had was maybe too complicated cause it involved 3 level depth child dependency, but it gave me straight wrong answer from the first glance misplaced the function in wrong context, instead trying to analyse it

Describe any alternatives that could be considered

no idea i canceled the sub and will wait for Q* probably

Additional context

No response

[kalanchan]

hey @MichalKurzewski thanks for the feedback!

Is this just happening on Claude 3 Opus? Were you able to get better responses from other LLM models? If you could post an example of your question to Cody we'd like to investigate further

[MichalKurzewski]

unfortunately its commercial product, cant send anything out. to add to that, the chat Opus handled it somehow, with somewhat better handling of failed trials (they made sense, but didnt work)

i can describe what i wanted and what happened in cody vs code:

• first i had the file opened and asked cody chat to explain the file, which it gave me an incorrect answer. i drilled down to the fact that it had incorrect scope analysing (file had 100 ish lines, but when I asked how many lines there it replied 48 or something similar, which was in the middle of function scope so totally wrong) after selecting everything in that file it worked ok on it.
• then I provided the context using @for file paths and asked the question again. It started to explain instead of attaching the previous prompt context.
• i made 3 attempts of providing full context and @ files in one prompt and it hallucinated so heavily i just couldnt look at it. the issue i had was in a buttonClickHandler in one of the component. cody suggested to move it into separate scope, which on the first glance would break everything.

thats it. i think ill just stick to web opus chat, and wait for q* doomsday

On Thu, 21 Mar 2024 at 22:07, Kalan @.***> wrote:

hey @MichalKurzewski https://github.com/MichalKurzewski thanks for the feedback!

Is this just happening on Claude 3 Opus? Were you able to get better responses from other LLM models? If you could post an example of your question to Cody we'd like to investigate further

— Reply to this email directly, view it on GitHub https://github.com/sourcegraph/cody/issues/3500#issuecomment-2013917043, or unsubscribe https://github.com/notifications/unsubscribe-auth/APKIPIGL6A3VSKDPZIVYAI3YZNKYZAVCNFSM6AAAAABFCG6TAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJTHEYTOMBUGM . You are receiving this because you were mentioned.Message ID: @.***>

[github-actions[bot]]

This issue is marked as stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed automatically in 5 days.