bug: this is bug AI model. When you ask 'which is greater between 9.11 and 9.8' and then 'which is greater between 9.8 and 9.11'

sourcegraph / cody

Type less, code more: Cody is an AI code assistant that uses advanced search and codebase context to help you write and fix code.

https://cody.dev

Apache License 2.0

2.71k stars 297 forks source link

bug: this is bug AI model. When you ask 'which is greater between 9.11 and 9.8' and then 'which is greater between 9.8 and 9.11' #5181

Open LvxlongSir opened 2 months ago

LvxlongSir commented 2 months ago

Version

latest, doesn't matter

Describe the bug

this is bug AI model. When you ask 'which is greater between 9.11 and 9.8' and then 'which is greater between 9.8 and 9.11'

Claude 3.5 Sonnet version it complete misleading itself to wrong answers...

Expected behavior

answer simple mathematic question: which is greater between 9.8 and 9.11

Additional context

No response

LvxlongSir commented 2 months ago

90DE11000FD3A9FB7E393A450E07C5B3

PriNova commented 2 months ago

Hello @LvxlongSir

Thank you for bringing this to attention. The issue mentioned is not directly related to Cody as a bug, but rather an inherent performance issue with some large language models. Currently, these models may or may not succeed in certain mathematical tests. As the models improve over time, they will become more adept at tasks requiring logic, mathematics, reasoning, and general problem-solving with greater precision.

Thank you.

github-actions[bot] commented 3 weeks ago

This issue is marked as stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed automatically in 5 days.