Please check the input prompt template

[dlibk]

Hi @adrielintan,

I have tried to run a new prompt template on trial split and record the actual prompt. Please see whether this is acceptable.

[[{"role": "system", "content": "Generate a background document from Wikipedia to support or refute the statement."}, {"role": "user", "content": "Statement: The Natural is a book about Roy Hobbs a natural southpaw boxer who goes on to win the heavyweight title from Boom Boom Mancini. \n\n"}]]

[[{"role": "system", "content": "Generate a background document from Wikipedia to support or refute the statement."}, {"role": "user", "content": "Statement: Johnny Depp has been in many successful films, including Black Mass, and is the 10th best paid actor in the world. \n\n"}]]

[[{"role": "system", "content": "Generate a background document from Wikipedia to support or refute the statement."}, {"role": "user", "content": "Statement: Ken Griffey Jr. was not at his dad's champion games because he had to stay home with a babysitter instead. \n\n"}]]

[[{"role": "system", "content": "Generate a background document from Wikipedia to support or refute the statement."}, {"role": "user", "content": "Statement: The film High Noon subverts gender norms of the time by having the woman rescue the man. \n\n"}]]

[[{"role": "system", "content": "Generate a background document from Wikipedia to support or refute the statement."}, {"role": "user", "content": "Statement: Pedro Alvares Cabral was a nobleman and awarded Knight in the King's Council. \n\n"}]]

If not, please kindly advise your expected prompt template by replying this issue in markdown style. Thanks for your effort.

[dlibk]

This change has not been commited yet BTW. Looking forward to reply.

[adrielintan]

The prompts you provided are what I expected. However, following up from our chat on WhatsApp, the prompts actually have several variations, especially for step 2. Please check the prompts in regular.jsonl. To be honest, I haven't thought about how we can do the prompting for part 2, but this is just my initial thought:

[{"role": "system", "content": "Provide a background document from Wikipedia to answer the given question."}, {"role": "user", "content": "%query%"}]
[{"role": "system", "content": "Generate a background document from Wikipedia to answer the given question."}, {"role": "user", "content": "%query%"}]]
[{"role": "system", "content": "Refer to the passage below and answer the following question with just one entity."}, {"role": "user", "content": "Passage: %background%\n\n Question: %query%\n\n The answer is:"}]]
[{"role": "system", "content": "Generate a background document from Wikipedia to support or refute the statement."}, {"role": "user", "content": "Statement: %query%"}]]
[{"role": "system", "content": "Based on the following background, decide if the claim is true or false"}, {"role": "user", "content": "Background: %background%\n\n Claim: $query$"}]]

Above is probably the prompt that we should use for the first five variations in regular.jsonl. For the last 2 variations (for type: dialogue system), I think we can just ignore it and not do any experiment on that dataset. What do you think?

[dlibk]

I can manually separate the prefix and statement and fill them into system and user respectively, which should be also aligned with original approach.

[adrielintan]

Please note that prompts with "task: step2" might not have any prefix, so we need to adjust it a little bit (e.g., the last prompt that I wrote).

[dlibk]

I plan to simply use str::split("\n\n", 1) to separate prefix and statement. If no prefix then content of system might be simply empty string. See the prompts of step2:

[
    [
        {
            "role": "system",
            "content": "I'm sorry, but the statement provided is inaccurate. \"The Natural\" is actually a novel written by Bernard Malamud in 1952, which tells the story of Roy Hobbs, a baseball player, not a boxer. The novel follows the life of Roy Hobbs, a talented baseball player with a mysterious past, as he navigates the world of professional baseball and faces various challenges and obstacles on his journey to success. The novel explores themes of ambition, love, and the corrupting influence of fame and success. It is considered a classic of American literature and has been adapted into a popular film starring Robert Redford as Roy Hobbs. In conclusion, \"The Natural\" is not about a southpaw boxer named Roy Hobbs winning the heavyweight title from Boom Boom Mancini, but rather a baseball player facing his own personal struggles and triumphs in the world of sports. "
        },
        {
            "role": "user",
            "content": " claim: The Natural is a book about Roy Hobbs a natural southpaw boxer who goes on to win the heavyweight title from Boom Boom Mancini. \n\n Is the claim true or false?"
        }
    ]
]
[
    [
        {
            "role": "system",
            "content": "Johnny Depp is an American actor, producer, and musician who has been in the entertainment industry for several decades. He has appeared in numerous successful films throughout his career, establishing himself as one of the most versatile and talented actors in Hollywood. One of Depp's notable roles was in the film \"Black Mass,\" released in 2015. In the film, he portrayed the notorious gangster Whitey Bulger, earning critical acclaim for his performance. \"Black Mass\" was a commercial success, grossing over $99 million worldwide. In addition to \"Black Mass,\" Johnny Depp has starred in a wide range of successful films, including the \"Pirates of the Caribbean\" franchise, \"Edward Scissorhands,\" \"Alice in Wonderland,\" and \"Charlie and the Chocolate Factory,\" among others. His ability to portray diverse characters and his unique acting style have contributed to his success in the film industry. As for his ranking as the 10th best-paid actor in the world, this information is subject to change based on various factors such as box office performance, endorsements, and other income sources. Johnny Depp has consistently been one of the highest-paid actors in Hollywood, thanks to his successful film career and lucrative endorsement deals. Overall, the statement that Johnny Depp has been in many successful films, including \"Black Mass,\" and is the 10th best-paid actor in the world is supported by his extensive filmography and his status as a highly sought-after actor in the industry. "
        },
        {
            "role": "user",
            "content": " claim: Johnny Depp has been in many successful films, including Black Mass, and is the 10th best paid actor in the world. \n\n Is the claim true or false?"
        }
    ]
]
[
    [
        {
            "role": "system",
            "content": "Ken Griffey Jr. is a former professional baseball player who played in Major League Baseball (MLB) for 22 seasons. He is widely regarded as one of the greatest baseball players of all time. Ken Griffey Jr.'s father, Ken Griffey Sr., was also a professional baseball player who had a successful career in MLB. There is no information or evidence available on Wikipedia or any other reputable source to support the statement that Ken Griffey Jr. was not present at his father's championship games because he had to stay home with a babysitter. In fact, Ken Griffey Jr. and his father have a close relationship, and Ken Griffey Jr. has often spoken about the influence his father had on his baseball career. Ken Griffey Jr. had a successful career in MLB, playing primarily for the Seattle Mariners and the Cincinnati Reds. He was known for his exceptional hitting and fielding abilities, as well as his charismatic personality. Ken Griffey Jr. was inducted into the Baseball Hall of Fame in 2016, further solidifying his status as one of the greatest players in the history of the sport. In conclusion, there is no credible information to support the statement that Ken Griffey Jr. was not present at his father's championship games because he had to stay home with a babysitter. On the contrary, Ken Griffey Jr. and his father have a strong bond, and Ken Griffey Jr. has always been supportive of his father's career in baseball. "
        },
        {
            "role": "user",
            "content": " claim: Ken Griffey Jr. was not at his dad's champion games because he had to stay home with a babysitter instead. \n\n Is the claim true or false?"
        }
    ]
]
[
    [
        {
            "role": "system",
            "content": "Background document from Wikipedia: \"High Noon\" is a 1952 American Western film directed by Fred Zinnemann and starring Gary Cooper and Grace Kelly. The film follows the story of a town marshal, Will Kane, who is forced to face a gang of outlaws alone after his town refuses to help him. One of the key aspects of the film that has been widely discussed is its portrayal of gender roles and norms. In the film, the character of Amy Fowler Kane, played by Grace Kelly, is depicted as a strong and independent woman who stands by her husband, Will Kane, throughout the ordeal. While it is true that Amy ultimately plays a crucial role in helping her husband, it is important to note that the film does not completely subvert traditional gender norms of the time. The character of Amy is shown as a supportive and loyal wife who stands by her husband in his time of need. However, it is ultimately Will Kane who takes on the role of the hero and faces the outlaws alone. The film does not completely reverse the traditional gender roles of the time, as it still portrays the man as the primary protector and savior. While Amy's actions in the film can be seen as empowering and progressive for the time, it is important to recognize that the overall narrative of \"High Noon\" still aligns with traditional gender norms. The film does not completely subvert these norms by having the woman rescue the man, but rather presents a more nuanced and complex portrayal of gender dynamics "
        },
        {
            "role": "user",
            "content": " claim: The film High Noon subverts gender norms of the time by having the woman rescue the man. \n\n Is the claim true or false?"
        }
    ]
]
[
    [
        {
            "role": "system",
            "content": "Background document from Wikipedia to support the statement: Pedro \u00c1lvares Cabral was indeed a nobleman and awarded Knight in the King's Council. According to historical records, Cabral was born in 1467 or 1468 into a noble family in Belmonte, Portugal. He was the third son of Fern\u00e3o \u00c1lvares Cabral, the Lord of Belmonte, and Isabel Gouveia. As a member of the Portuguese nobility, Cabral was raised in a privileged environment and received a good education. In addition to his noble background, Cabral was also recognized for his service to the Portuguese crown. He was appointed as a Knight in the King's Council by King Manuel I of Portugal, who ruled from 1495 to 1521. This title was a prestigious honor that was bestowed upon individuals who had demonstrated loyalty and service to the king. Cabral's status as a nobleman and Knight in the King's Council played a significant role in his career as a navigator and explorer. In 1500, he led a fleet of ships on a voyage to India, but instead, he accidentally discovered Brazil. This event marked the beginning of Portuguese colonization in the region and established Cabral as a key figure in the Age of Discovery. Overall, the historical evidence supports the statement that Pedro \u00c1lvares Cabral was a nobleman and awarded Knight in the King's Council, highlighting his privileged background and prestigious title within the Portuguese royal court. "
        },
        {
            "role": "user",
            "content": " claim: Pedro Alvares Cabral was a nobleman and awarded Knight in the King's Council. \n\n Is the claim true or false?"
        }
    ]
]

[adrielintan]

I don't think it's a good idea to put the documents in the "system". From a quick read on Google, "system" is used to give a personality to the model. So, any data provided here is weakly referred. Also, some sources also mention about the reluctancy of the model to reveal what's provided in the "system" to the output. So, the above prompts may not give good replies considering the model needs to retrieve the information from the "system".

[dlibk]

I don't think it's a good idea to put the documents in the "system". From a quick read on Google, "system" is used to give a personality to the model. So, any data provided here is weakly referred. Also, some sources also mention about the reluctancy of the model to reveal what's provided in the "system" to the output. So, the above prompts may not give good replies considering the model needs to retrieve the information from the "system".

I can open another issue for a better template. This issue and related PR will remain open before there is a conclusion.

We can let one of our member to take care of this. They can try different prompt on GUI website of OpenAI and select a good prompt template.

I need to leave for course for a while. I will open another issue for template at night if there's no additional suggestion for me.

[adrielintan]

Sorry for the late reply. I think I said it a bit vaguely. In my opinion, the prompt given by the original author is fine. However, because davinci is already deprecated, we need to adapt their prompt in gpt right. And I believe the most recent prompts you provided (for step 2) are quite problematic because the generated documents are put in the "system" instead of "user". So, I think just moving the documents to the "user" is fine.

[adrielintan]

That said, the new issue is probably unnecessary?

[dlibk]

I will revert the latest commit later and switch the prompt back to the naive approach (see the json below). Please approve the pull request #2 after you find the prompt template is satisfactory. This issue will be automatically closed.

Should you want to make any change, please give a clear instruction of the template in JSON format (with some necessary place holder) and avoid using pure natural language. I will probably not commit any code changes based on pure natural language instruction in the future (if it's still about prompt template). It's not necessary for you to explain the reason to me -- @Yuanky123 is taking care about this.

If you are not sure about the template, please discuss with @Yuanky123 under #3 . Also, give the new template in JSON format and mention me.

However, I don't think the issue #3 is unnecessary because other members also need the contribution credits. We need to distribute some task even if you don't think it's useful for you.

BTW, I would suggest we distribute some tasks to other members otherwise they may get worried. I think Kangyu can be your assistant for experiment. And we can have another part "Related Works" which is isolated from other parts, and we can assign one member to do that without much communication with others (should be efficient then).

{
  "role": "system", "content": "You are a helpful assistant",
  "role": "user", "content": prefix + statement given by original work
}

Also, we need to spend some time on report now. I will take care about this and open another issue for sections distribution later if you don't have objection on that.

[adrielintan]

In that case please follow the following json formatting:

//for {"type": "question answering", "task": "step1", "pid": 1}
[
        {"role": "system", "content": "Provide a background document from Wikipedia to answer the given question."},
        {"role": "user", "content": "%query%"}
]

//for {"type": "question answering", "task": "step1", "pid": 2}
[
        {"role": "system", "content": "Generate a background document from Wikipedia to answer the given question."},
        {"role": "user", "content": "%query%"}
]

//for {"type": "question answering", "task": "step2", "pid": 1}
[
        {"role": "system", "content": "Refer to the passage below and answer the following question with just one entity."},
        {"role": "user", "content": "Passage: %background%\n\n Question: %query%\n\n The answer is:"}
]

//for {"type": "fact checking", "task": "step1", "pid": 1}
[
        {"role": "system", "content": "Generate a background document from Wikipedia to support or refute the statement."},
        {"role": "user", "content": "Statement: %query%"}
]

//for {"type": "fact checking", "task": "step2", "pid": 1}
[
        {"role": "system", "content": "Based on the following background, decide if the claim is true or false"},
        {"role": "user", "content": "Background: %background%\n\n Claim: $query$"}
]