Chainlit debugging best practices?

[asmith26]

Hi there, thanks very much for sharing this library, it looks great!

I'm trying out the PDF-QA example with a local LLM, but I keep getting: ERROR: The prompt size exceeds the context window size and cannot be processed.. I'm trying to identify/edit the prompt, but I can't find it?

Not sure if you can help with this specific question or chainlit debugging best practices more generally?

Thanks!

[willydouhard]

Hey, what is the context window size of your local LLM? Make sure that the max_tokens_limit parameter is not exceeding it.

[asmith26]

Thanks for the info and quick response.

So I also get an error of LLaMA ERROR: The prompt is2906tokens and the context window is2048!, so I tried changing the setting to 2048 but unfortunately I still seem to be getting errors.

I've added my full changes to https://github.com/Chainlit/cookbook/pull/6/files in case you notice something I'm doing wrong (I'm trying to use gpt4all for the local chat model, but I'm open to using any local model if you have a better example).

Many thanks for any help!

[willydouhard]

Maybe you could change the chunk size int the text splitter and the k parameter here to limit the number of documents returned by a search. Your context window is 2048 tokens so maybe a chunk size of 500 and k=2?

[asmith26]

Thanks for the response and suggestions, using a different model seems to have fixed this specific issue.

Interestingly when I click on the little pencil to open the Prompt Playground, the prompt is different to what I expect - e.g. if I upload a dataset of Alice in Wonderland and ask the question "Who wrote Alice’s Adventures in Wonderland?", the prompt also seems to include a lot of other text/context:

Given the following extracted parts of a long document and a question, create a final answer with references ("SOURCES"). 
If you don't know the answer, just say that you don't know. Don't try to make up an answer.
ALWAYS return a "SOURCES" part in your answer.

QUESTION: Which state/country's law governs the interpretation of the contract?
=========
Content: This Agreement is governed by English law and the parties submit to the exclusive jurisdiction of the English courts in  relation to any dispute (contractual or non-contractual) concerning this Agreement save that either party may apply to any court for an  injunction or other relief to protect its Intellectual Property Rights.
Source: 28-pl
Content: No Waiver. Failure or delay in exercising any right or remedy under this Agreement shall not constitute a waiver of such (or any other)  right or remedy.

11.7 Severability. The invalidity, illegality or unenforceability of any term (or part of a term) of this Agreement shall not affect the continuation  in force of the remainder of the term (if any) and this Agreement.

11.8 No Agency. Except as expressly stated otherwise, nothing in this Agreement shall create an agency, partnership or joint venture of any  kind between the parties.

11.9 No Third-Party Beneficiaries.
Source: 30-pl
Content: (b) if Google believes, in good faith, that the Distributor has violated or caused Google to violate any Anti-Bribery Laws (as  defined in Clause 8.5) or that such a violation is reasonably likely to occur,
Source: 4-pl
=========
FINAL ANSWER: This Agreement is governed by English law.
SOURCES: 28-pl

QUESTION: What did the president say about Michael Jackson?
=========
Content: Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  

Last year COVID-19 kept us apart. This year we are finally together again. 

Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. 

With a duty to one another to the American people to the Constitution. 

And with an unwavering resolve that freedom will always triumph over tyranny. 

Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. 

He thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. 

He met the Ukrainian people. 

From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world. 

Groups of citizens blocking tanks with their bodies. Everyone from students to retirees teachers turned soldiers defending their homeland.
Source: 0-pl
Content: And we won’t stop. 

We have lost so much to COVID-19. Time with one another. And worst of all, so much loss of life. 

Let’s use this moment to reset. Let’s stop looking at COVID-19 as a partisan dividing line and see it for what it is: A God-awful disease.  

Let’s stop seeing each other as enemies, and start seeing each other for who we really are: Fellow Americans.  

We can’t change how divided we’ve been. But we can change how we move forward—on COVID-19 and other issues we must face together. 

I recently visited the New York City Police Department days after the funerals of Officer Wilbert Mora and his partner, Officer Jason Rivera. 

They were responding to a 9-1-1 call when a man shot and killed them with a stolen gun. 

Officer Mora was 27 years old. 

Officer Rivera was 22. 

Both Dominican Americans who’d grown up on the same streets they later chose to patrol as police officers. 

I spoke with their families and told them that we are forever in debt for their sacrifice, and we will carry on their mission to restore the trust and safety every community deserves.
Source: 24-pl
Content: And a proud Ukrainian people, who have known 30 years  of independence, have repeatedly shown that they will not tolerate anyone who tries to take their country backwards.  

To all Americans, I will be honest with you, as I’ve always promised. A Russian dictator, invading a foreign country, has costs around the world. 

And I’m taking robust action to make sure the pain of our sanctions  is targeted at Russia’s economy. And I will use every tool at our disposal to protect American businesses and consumers. 

Tonight, I can announce that the United States has worked with 30 other countries to release 60 Million barrels of oil from reserves around the world.  

America will lead that effort, releasing 30 Million barrels from our own Strategic Petroleum Reserve. And we stand ready to do more if necessary, unified with our allies.  

These steps will help blunt gas prices here at home. And I know the news about what’s happening can seem alarming. 

But I want you to know that we are going to be okay.
Source: 5-pl
Content: More support for patients and families. 

To get there, I call on Congress to fund ARPA-H, the Advanced Research Projects Agency for Health. 

It’s based on DARPA—the Defense Department project that led to the Internet, GPS, and so much more.  

ARPA-H will have a singular purpose—to drive breakthroughs in cancer, Alzheimer’s, diabetes, and more. 

A unity agenda for the nation. 

We can do this. 

My fellow Americans—tonight , we have gathered in a sacred space—the citadel of our democracy. 

In this Capitol, generation after generation, Americans have debated great questions amid great strife, and have done great things. 

We have fought for freedom, expanded liberty, defeated totalitarianism and terror. 

And built the strongest, freest, and most prosperous nation the world has ever known. 

Now is the hour. 

Our moment of responsibility. 

Our test of resolve and conscience, of history itself. 

It is in this moment that our character is formed. Our purpose is found. Our future is forged. 

Well I know this nation.
Source: 34-pl
=========
FINAL ANSWER: The president did not mention Michael Jackson.
SOURCES:

QUESTION: Who wrote Alice’s Adventures in Wonderland?
=========
Content: Alice’s Adventures in Wonderland

by Lewis Carroll

THE MILLENNIUM FULCRUM EDITION 3.0

Contents

 CHAPTER I.     Down the Rabbit-Hole
 CHAPTER II.    The Pool of Tears
 CHAPTER III.   A Caucus-Race and a Long Tale
 CHAPTER IV.    The Rabbit Sends in a Little Bill
 CHAPTER V.     Advice from a Caterpillar
 CHAPTER VI.    Pig and Pepper
 CHAPTER VII.   A Mad Tea-Party
 CHAPTER VIII.  The Queen’s Croquet-Ground
 CHAPTER IX.    The Mock Turtle’s Story
 CHAPTER X.     The Lobster Quadrille
 CHAPTER XI.    Who Stole the Tarts?
 CHAPTER XII.   Alice’s Evidence

CHAPTER I.
Down the Rabbit-Hole

Alice was beginning to get very tired of sitting by her sister on the
bank, and of having nothing to do: once or twice she had peeped into
the book her sister was reading, but it had no pictures or
conversations in it, “and what is the use of a book,” thought Alice
“without pictures or conversations?”
Source: source_0

Content: “Exactly so,” said the Hatter: “as the things get used up.”

“But what happens when you come to the beginning again?” Alice ventured
to ask.

“Suppose we change the subject,” the March Hare interrupted, yawning.
“I’m getting tired of this. I vote the young lady tells us a story.”

“I’m afraid I don’t know one,” said Alice, rather alarmed at the
proposal.

“Then the Dormouse shall!” they both cried. “Wake up, Dormouse!” And
they pinched it on both sides at once.

The Dormouse slowly opened his eyes. “I wasn’t asleep,” he said in a
hoarse, feeble voice: “I heard every word you fellows were saying.”

“Tell us a story!” said the March Hare.

“Yes, please do!” pleaded Alice.

“And be quick about it,” added the Hatter, “or you’ll be asleep again
before it’s done.”

“Once upon a time there were three little sisters,” the Dormouse began
in a great hurry; “and their names were Elsie, Lacie, and Tillie; and
they lived at the bottom of a well—”
Source: source_100

Content: “Well! I’ve often seen a cat without a grin,” thought Alice; “but a
grin without a cat! It’s the most curious thing I ever saw in my life!”

She had not gone much farther before she came in sight of the house of
the March Hare: she thought it must be the right house, because the
chimneys were shaped like ears and the roof was thatched with fur. It
was so large a house, that she did not like to go nearer till she had
nibbled some more of the lefthand bit of mushroom, and raised herself
to about two feet high: even then she walked up towards it rather
timidly, saying to herself “Suppose it should be raving mad after all!
I almost wish I’d gone to see the Hatter instead!”

CHAPTER VII.
A Mad Tea-Party
Source: source_91

Content: “But then,” thought Alice, “shall I _never_ get any older than I am
now? That’ll be a comfort, one way—never to be an old woman—but
then—always to have lessons to learn! Oh, I shouldn’t like _that!_”

“Oh, you foolish Alice!” she answered herself. “How can you learn
lessons in here? Why, there’s hardly room for _you_, and no room at all
for any lesson-books!”

And so she went on, taking first one side and then the other, and
making quite a conversation of it altogether; but after a few minutes
she heard a voice outside, and stopped to listen.

“Mary Ann! Mary Ann!” said the voice. “Fetch me my gloves this moment!”
Then came a little pattering of feet on the stairs. Alice knew it was
the Rabbit coming to look for her, and she trembled till she shook the
house, quite forgetting that she was now about a thousand times as
large as the Rabbit, and had no reason to be afraid of it.
Source: source_48
=========
FINAL ANSWER:
Lewis Carroll

What prompt template does Chainlit use? Is the additional context/questions coming from Chainlit (or possibly Langchain)?

Many thanks again!

[willydouhard]

This is coming from langchain. They prefix the prompt with a few examples (this is called few shot prompting) to show the LLM what the expected format is.

This is the default prompt for this chain, you can tweak it if you need to.

[asmith26]

This is the default prompt for this chain, you can tweak it if you need to.

Thanks, sorry do you have any useful links/example regarding how to change this? Thanks again

[willydouhard]

Maybe this can help https://stackoverflow.com/questions/76240871/how-do-i-add-memory-to-retrievalqa-from-chain-type-or-how-do-i-add-a-custom-pr

[dmi-vsalaria]

how to debug llm when using chainlit?

[willydouhard]

We are going to release a new version of the prompt playground tomorrow which will help a lot for debugging