[BUG] Prompt template didn't allow providing details of transcribed lesson

[davide445]

Describe the bug Used STT to transcribe a lesson (apologize it's in Italian, it's a lesson about physics education). Testing how to create an assistant answering questions. It answers only to general summary questions but not about details like "what talked the little dragon Spike about?" or "how is discussed the peltier cell?".

The very same document with the same prompt works fine in openai assistant and in poe with Claude 3, able to provide details about these topics.

I have tried changing the loader in txt, docx, text, changing the embedding model in openai, cohere, the vectordb to drant, pinecone, inmemory, changing text encoding to UTF-8 , the chunking from no setting to 1000 to 500, but the answer is always "I have no information" and provide me info from his own training instead of using the text.

Even changing the LLM to GPT3.5 Turbo 0.125 and using the very same prompt used in the Assistant replicate the same problem.

Using another document like the attached habilis (changing the prompt to like "you are an expert teacher discussing about paleonthology") seems is working well, but I can't understand why.

My understanding of prompt engineering is of course very limited, but this is really a basic use and can't find a reason why cannot work

To Reproduce

1. import attached flow
2. translate the prompt in English
3. import the attached STT_lesson.txt
4. upsert
5. ask about what the little dragon spike talk about

Expected behavior This is openai assistant answer (in Italian, I have translated in English here) to the question "what talks about the little dragon spike" The little dragon Spike is the main character in a story that is used to introduce the concept of warmth to preschool children. Spike is a happy little dragon who lives with his parents on a mountaintop and spends his time flying and observing the world from above. In the story, Spike comes into contact with two little ghosts, Anger and Heat, who make him experience feelings of anger and heat. Through this narrative, parallels are drawn between anger and heat, with the goal of helping children understand the concept of heat as an invisible entity distinct from the simple feeling of hot or cold things

Screenshots

Flow See attached

Setup

• npx flowise start, docker run -p 6333:6333 -p 6334:6334 -v $(pwd)/qdrant_storage:/qdrant/storage:z qdrant/qdrant

• Flowise Version 1.6.6

• OS: Linux

• Browser Firefox

Additional context Add any other context about the problem here. STT_lesson.txt Habilis.txt Lesson assistant Chatflow.json

[toi500]

So, you are looking somehing like this?

[davide445]

Of course, I dont know how you obtained this,if just changing the prompt or something else

[toi500]

Your flow is too complex for the task you are looking for. I used a simple version, using the your Response Prompt in italian:

Change the embedding and chat model nodes to OpenAI if you dont have access to Gemini, which is better that GPT4 by the way :)

Chatflow.json

[davide445]

This is sure good to know, but I need to have a prompt due I want to give the chat model instructions about how to behave for specific topics. Have you eventually tried any other prompt.

[toi500]

You instructions go here:

Here is another reply:

For multi topics you can use the Multi Retrieval QA Chain, from the Marketplace.

[davide445]

And in which node are the rephrase and response prompt? I have never seen them

[davide445]

I have found it, it's in the Conversational Retrieval QA Chain parameters, I need to test this. I will be anyway happy to know why this is not working in the Prompt Template I was using. Is requiring a different prompting style? Sooner or later I suppose I will need this one, due for other projects I was testing the Tool chain.

[davide445]

I't bizarre, at least in my opinion. If I leave the Rephrase prompt of the Conversational Retrieval QA Chain in English as your all seems working fine. If I translate it in Italian (as was my first activity) I got the same problems I get with the Chat Prompt Template. To have a test I tried changing to English the Chat Prompt Template system prompt, but on the contrary didn't got positive effects. Wanted to ask if there is any known reason. I found on langchain documentation info about the Chat Prompt Template, but not about the Conversational Retrieval QA Chain, so that I can't understand why this behavior.

[toi500]

Then it is a problem with OpenAI / Mistral that does not work fine in Italian. Note that I used Gemini 1.5 in my test and worked fine.

[davide445]

I'm using Claude 3 Haiku, perfectly capable in Italian

[davide445]

Reworking a bit the Italian version of the Rephrase prompt this is working also, probably my original translation was not good enough. I didn't have a Google Gemini 1.5 access, but testing Claude 3 Haiku, Sonnet, GPT 3.5 Turbo 0125, Cohere Command-r the latest seems the better one with this kind data. Something surprise me, due normally Claude 3 is the best one, but there is always something to learn.