[Feature Request]: Adding an actual System prompt

[yasergsi]

Feature Description

Currently the system_prompt is taken as a parameter in the llama_index.llm_predictor.base.LLMPredictor Class but when llama-Index queries the OpenAI API it just adds this system prompt just before the User query i.e. the role defined is user when querying the OpenAI API whereas there should be a separate role i.e. SYSTEM with the content as the defined custom System prompt.

I am trying to use this as I am implementing Query Engine. Also I think the System Prompt should be a parameter while creating Query Engine using low level coding i.e. by utilizing the class RetrieverQueryEngine

Reason

Not sure what is stopping Llama-index from doing this (it should be pretty simple), the current implementation of system prompt is a buggy way of doing it.

Value of Feature

This will be of immense value since GPT gives more importance to system prompts. In the current implementation the instructions in the system prompt is not given too much priority since it is part of user query.

[logan-markewich]

@yasergsi Not sure what you mean here.

The system prompt in LLMPredictor is added as the system role https://github.com/jerryjliu/llama_index/blob/3f6d982c3e2dae71ee240fc2323b5111bd2fd4f7/llama_index/llm_predictor/base.py#L228C14-L228C14

If you want further customization i.e. in the query engine, you can configure the prompt templates https://gpt-index.readthedocs.io/en/stable/examples/customization/prompts/chat_prompts.html

If you have an idea for improving this, I definitely welcome a PR :)

[logan-markewich]

And of course, this is all specific to query engines.

The chat-engines/agents support system messages, inserting the system message at the start of the conversation history at every chat message

[yasergsi]

Thanks for the response @logan-markewich, but when I see the logs I see the SYSTEM PROMPT as a part of USER PROMPT only as shown in the example below.

My Code

SYSTEM_PROMPT = (
    "You are QABot - a chatbot that answers users query based on custom knowledge base provided by the user."
    "Always output responses in markdown for better readability."
)

text_qa_msgs = [
    ChatMessage(
        role=MessageRole.SYSTEM,
        content=SYSTEM_PROMPT,
    ),
    ChatMessage(
        role=MessageRole.USER,
        content=CUSTOM_TEXT_QA_PROMPT_TMPL,
    ),
]

text_qa_template = ChatPromptTemplate(text_qa_msgs)

response_synthesizer = get_response_synthesizer(
    service_context=service_context,
    text_qa_template=text_qa_template,
    response_mode=ResponseMode.SIMPLE_SUMMARIZE,
    streaming=True
)

query_engine = RetrieverQueryEngine.from_args(
    retriever=retriever,
    response_synthesizer=response_synthesizer,
    response_mode=ResponseMode.SIMPLE_SUMMARIZE,
    text_qa_template=text_qa_template,
    service_context=service_context,
    streaming=True,
)
query_text = "How many sessions on the first day?"
response = query_engine.query(query_text)

The 'API call' found in logs

DEBUG:openai:api_version=None data='{"messages": [{"role": "user", "content": "system: You are QABot - a chatbot that answers users query based on custom knowledge base provided by the user.Always output responses in markdown for better readability.\\nuser: Sharing that the current date is 2023-09-05 and the current time is 07:41:40, which you can refer to as a point of context to answer any time-related questions.Context information is below.\\n---------------------\\nobject_id: 241ede4c-401b-4e1c-b89d-071e4cebfb34\\n\\nThemes to be covered in the agenda include:\\n\\nFuture of MICE\\n\\nTravel\\nTechnology\\n\\nTravel \\nDistribution\\n\\nOTAs &\\nIntermediaries\\n\\nDirect\\nBooking\\n\\nCorporate \\nTravel\\n\\nHotels &\\nAlternative \\nAccommodation\\n\\nVacations &\\nTours\\n\\nDestination\\nMarketing\\n\\nLeadership &\\nInspirations\\n\\nAirlines & Air\\nTravel\\n\\nMuslim Travel\\n\\nCruise\\n\\nPayment &\\nAlternative\\nCurrencies\\n\\nGlobal\\nConsumer\\nTrends\\n\\nSocial Media &\\nInfluencer\\nMarketing\\n\\n|Club\\n---------------------\\nNow, respond to the message given below\\n---------------------\\nHow many sessions on the first day?\\n---------------------\\n\\nassistant: "}], "model": "gpt-4", "temperature": 0, "stream": true}' message='Post details'

I think the expected API call should have an additional key as "role":"system", "content":<my-system-prompt>

API call was similar in the logs when previously I added the system_prompt in llama_index.llm_predictor.base.LLMPredictor

Note: I'm using the latest version as of today i.e. 0.8.20

[logan-markewich]

@yasergsi hmm super weird. Lets see if I can debug why that's happening

[logan-markewich]

@yasergsi what is your service context like? Which LLM are you using?

You should use the LLM classes from llama-index to get full proper functionality. Running your exact code, I see this in my debug logs

DEBUG:openai:api_version=None data='{"messages": [{"role": "system", "content": "You are QABot - a chatbot that answers users query based on custom knowledge base provided by the user.Always output responses in markdown for better readability."}, {"role": "user", "content": "Context: What I Worked On\\n\\nFebruary 2021\\n\\nBefore college the two main things I worked on, outside of school, were writing and programming.I didn\'t write essays.I wrote what beginning writers were supposed to write then, and probably still are: short stories.My stories were awful.They had hardly any plot, just characters with strong feelings, which I imagined made them deep.The first programs I tried writing were on the IBM 1401 that our school district used for what was then called \\"data processing.\\"This was in 9th grade, so I was 13 or 14.The school district\'s 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it.It was like a mini Bond villain\'s lair down there, with all these alien-looking machines \\u2014 CPU, disk drives, printer, card reader \\u2014 sitting up on a raised floor under bright fluorescent lights.The language we used was an early version of Fortran.You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it.The result would ordinarily be to print something on the spectacularly loud printer.I was puzzled by the 1401.I couldn\'t figure out what to do with it.And in retrospect there\'s not much I could have done with it.The only form of input to programs was data stored on punched cards, and I didn\'t have any data stored on punched cards.The only other option was to do things that didn\'t rely on any input, like calculate approximations of pi, but I didn\'t know enough math to do anything interesting of that type.So I\'m not surprised I can\'t remember any programs I wrote, because they can\'t have done much.My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn\'t.On a machine without time-sharing, this was a social as well as a technical error, as the data center manager\'s expression made clear.With microcomputers, everything changed.Now you could have a computer sitting right in front of you, on a desk, that could respond to your keystrokes as it was running instead of just churning through a stack of punch cards and then stopping.[1]\\n\\nThe first of my friends to get a microcomputer built it himself.It was sold as a kit by Heathkit.I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer.Computers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980.The gold standard then was the Apple II, but a TRS-80 was good enough.This was when I really started programming.I wrote simple games, a program to predict how high my model rockets would fly, and a word processor that my father used to write at least one book.There was only room in memory for about 2 pages of text, so he\'d write 2 pages at a time and then print them out, but it was a lot better than a typewriter.Though I liked programming, I didn\'t plan to study it in college.In college I was going to study philosophy, which sounded much more powerful.It seemed, to my naive high school self, to be the study of the ultimate truths, compared to which the things studied in other fields would be mere domain knowledge.What I discovered when I got to college was that the other fields took up so much of the space of ideas that there wasn\'t much left for these supposed ultimate truths.All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.I couldn\'t have put this into words when I was 18.All I knew at the time was that I kept taking philosophy courses and they kept being boring.So I decided to switch to AI.AI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU.I haven\'t tried rereading The Moon is a Harsh Mistress, so I don\'t know how well it has aged, but when I read it I was drawn entirely into its world.It seemed only a matter of time before we\'d have Mike, and when I saw Winograd using SHRDLU, it seemed like that time would be a few years at most.All you had to do was teach SHRDLU more words.There weren\'t any classes in AI at Cornell then, not even graduate classes, so I started trying to teach myself.Which meant learning Lisp, since in those days Lisp was regarded as the language of AI.\\nThere, right on the wall, was something you could make that would last.Paintings didn\'t become obsolete.Some of the best ones were hundreds of years old.And moreover this was something you could make a living doing.Not as easily as you could by writing software, of course, but I thought if you were really industrious and lived really cheaply, it had to be possible to make enough to survive.And as an artist you could be truly independent.You wouldn\'t have a boss, or even need to get research funding.I had always liked looking at paintings.Could I make them?I had no idea.I\'d never imagined it was even possible.I knew intellectually that people made art \\u2014 that it didn\'t just appear spontaneously \\u2014 but it was as if the people who made it were a different species.They either lived long ago or were mysterious geniuses doing strange things in profiles in Life magazine.The idea of actually being able to make art, to put that verb before that noun, seemed almost miraculous.That fall I started taking art classes at Harvard.Grad students could take classes in any department, and my advisor, Tom Cheatham, was very easy going.If he even knew about the strange classes I was taking, he never said anything.So now I was in a PhD program in computer science, yet planning to be an artist, yet also genuinely in love with Lisp hacking and working away at On Lisp.In other words, like many a grad student, I was working energetically on multiple projects that were not my thesis.I didn\'t see a way out of this situation.I didn\'t want to drop out of grad school, but how else was I going to get out?I remember when my friend Robert Morris got kicked out of Cornell for writing the internet worm of 1988, I was envious that he\'d found such a spectacular way to get out of grad school.Then one day in April 1990 a crack appeared in the wall.I ran into professor Cheatham and he asked if I was far enough along to graduate that June.I didn\'t have a word of my dissertation written, but in what must have been the quickest bit of thinking in my life, I decided to take a shot at writing one in the 5 weeks or so that remained before the deadline, reusing parts of On Lisp where I could, and I was able to respond, with no perceptible delay \\"Yes, I think so.I\'ll give you something to read in a few days.\\"I picked applications of continuations as the topic.In retrospect I should have written about macros and embedded languages.There\'s a whole world there that\'s barely been explored.But all I wanted was to get out of grad school, and my rapidly written dissertation sufficed, just barely.Meanwhile I was applying to art schools.I applied to two: RISD in the US, and the Accademia di Belli Arti in Florence, which, because it was the oldest art school, I imagined would be good.RISD accepted me, and I never heard back from the Accademia, so off to Providence I went.I\'d applied for the BFA program at RISD, which meant in effect that I had to go to college again.This was not as strange as it sounds, because I was only 25, and art schools are full of people of different ages.RISD counted me as a transfer sophomore and said I had to do the foundation that summer.The foundation means the classes that everyone has to take in fundamental subjects like drawing, color, and design.Toward the end of the summer I got a big surprise: a letter from the Accademia, which had been delayed because they\'d sent it to Cambridge England instead of Cambridge Massachusetts, inviting me to take the entrance exam in Florence that fall.This was now only weeks away.My nice landlady let me leave my stuff in her attic.I had some money saved from consulting work I\'d done in grad school; there was probably enough to last a year if I lived cheaply.Now all I had to do was learn Italian.Only stranieri (foreigners) had to take this entrance exam.In retrospect it may well have been a way of excluding them, because there were so many stranieri attracted by the idea of studying art in Florence that the Italian students would otherwise have been outnumbered.I was in decent shape at painting and drawing from the RISD foundation that summer, but I still don\'t know how I managed to pass the written exam.I remember that I answered the essay question by writing about Cezanne, and that I cranked up the intellectual level as high as I could to make the most of my limited vocabulary.[2]\\n\\nI\'m only up to age 25 and already there are such conspicuous patterns.Here I was, yet again about to attend some august institution in the hopes of learning about some prestigious subject, and yet again about to be disappointed.\\n\\nQuestion: How many sessions on the first day?\\n\\nAnswer: "}], "stream": false, "model": "gpt-3.5-turbo", "temperature": 0.1, "max_tokens": null}' message='Post details'

Full code:

from llama_index import ServiceContext
from llama_index.response_synthesizers import get_response_synthesizer, ResponseMode
from llama_index.query_engine import RetrieverQueryEngine
from llama_index.llms import MessageRole, ChatMessage, OpenAI
from llama_index.prompts import ChatPromptTemplate

llm = OpenAI(model="gpt-3.5-turbo", temperature=0.1)
service_context = ServiceContext.from_defaults(llm=llm)

SYSTEM_PROMPT = (
    "You are QABot - a chatbot that answers users query based on custom knowledge base provided by the user."
    "Always output responses in markdown for better readability."
)

text_qa_msgs = [
    ChatMessage(
        role=MessageRole.SYSTEM,
        content=SYSTEM_PROMPT,
    ),
    ChatMessage(
        role=MessageRole.USER,
        content="Context: {context_str}\n\nQuestion: {query_str}\n\nAnswer: ",
    ),
]

text_qa_template = ChatPromptTemplate(text_qa_msgs)

response_synthesizer = get_response_synthesizer(
    service_context=service_context,
    text_qa_template=text_qa_template,
    response_mode=ResponseMode.SIMPLE_SUMMARIZE,
    streaming=True
)

query_engine = RetrieverQueryEngine.from_args(
    retriever=index.as_retriever(),
    response_synthesizer=response_synthesizer,
    response_mode=ResponseMode.SIMPLE_SUMMARIZE,
    text_qa_template=text_qa_template,
    service_context=service_context,
    streaming=True,
)
query_text = "How many sessions on the first day?"
response = query_engine.query(query_text)
response.print_response_stream()

[yasergsi]

Ohh Alright !! Thanks @logan-markewich The only difference was the one import statement for OpenAI

I was using from langchain import OpenAI (from the older llama-Index documentations maybe)

Changing it to from llama_index.llms import OpenAI makes it work correctly