bitcoinsearch / chat-btc

A conversational AI based on technical bitcoin sources
https://chat.bitcoinsearch.xyz/
MIT License
14 stars 13 forks source link

Failure generating response #84

Open Emmanuel-Develops opened 1 month ago

Emmanuel-Develops commented 1 month ago

problem

I'm not able to get an answer for "silent payments" but other queries work Screenshot_20240514_032159_Samsung_InternetScreenshot 2024-05-14 at 12 45 25 PM

deductions

kouloumos commented 1 month ago

Possible explanation: Token limit cuts off the remaining sources as they were too large

Token limit is 7000 characters. The source that is included (github) is ~273 tokens based on this tool.

edit: Okay, my bad, so what you are saying is that the remaining sources add up to more than 7000 tokens. Do we have access to what those sources are? So that we can verify it?

Emmanuel-Develops commented 1 month ago

edit: Okay, my bad, so what you are saying is that the remaining sources add up to more than 7000 tokens. Do we have access to what those sources are? So that we can verify it?

Yeah, but more like the next relevant source in addition with the most relevant (the single source shown) adds up to more than 7000 tokens.

Yeah, on local you can disable the tokenEnforcer and see the sources used or log the sources before then. I ran the query again and these are the results with tokenEnforcer disabled. Note: the order of the sources may not be exactly what the user got the first time

[0]: https://bitcoinops.org/en/topics/silent-payments [1]: https://btctranscripts.com/bitcoin-explained/silent-payments [2]: https://btctranscripts.com/bitcoin-core-dev-tech/2023-04/2023-04-26-silent-payments [3]: https://btctranscripts.com/bitcoin-design/learning-bitcoin-and-design/silent-payments [4]: https://delvingbitcoin.org/t/about-the-wg-silent-payments-category/876 [5]: https://btctranscripts.com/bitcoin-explained/accounts-for-bitcoin-easypaysy [6]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-May/020519.html [7]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-August/021844.html [8]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-August/020885.html [9]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-August/021864.html

source [1] token count is over 11,000.

With token enforcer you have this (this is from the same prompt)

{ tokenLength: 39392 }
{ tokenLength: 36922 }
{ tokenLength: 36616 }
{ tokenLength: 32113 }
{ tokenLength: 31568 }
{ tokenLength: 24684 }
{ tokenLength: 24558 }
{ tokenLength: 11179 }
{ tokenLength: 10278 }
{ tokenLength: 299 }
enforceToken: 779.628ms

so it keeps removing sources from least relevant till it fits into the context window. Therefore explains the single source [0] that was selected.