Spring EM issue - Githubissues

codecodebear commented 1 month ago

Hi, I get extremely low EM for Spring.

I set up Spring following configurations in reproduction guideline.

for Spring, the metrics when evaluated with NQ are:

em: 0.00443213296398892
f1: 0.13373634365821715
acc: 0.5548476454293629
precision: 0.07915875093799442
recall: 0.6200184672206834

and the first item in intermediate data is:

{
        "id": "test_0",
        "question": "who got the first nobel prize in physics",
        "golden_answers": [
            "Wilhelm Conrad R\u00f6ntgen"
        ],
        "output": {
            "retrieval_result": [
                {
                    "id": "16315106",
                    "contents": "\"Nobel Prize in Physics\"\nNobel Prize in Physics The Nobel Prize in Physics () is a yearly award given by the Royal Swedish Academy of Sciences for those who have made the most outstanding contributions for mankind in the field of physics. It is one of the five Nobel Prizes established by the will of Alfred Nobel in 1895 and awarded since 1901; the others being the Nobel Prize in Chemistry, Nobel Prize in Literature, Nobel Peace Prize, and Nobel Prize in Physiology or Medicine. The first Nobel Prize in Physics was awarded to physicist Wilhelm R\u00f6ntgen in recognition of the extraordinary services he",
                    "score": 12.15781021118164
                },
                {
                    "id": "16315118",
                    "contents": "\"Nobel Prize in Physics\"\nreceive a diploma, a medal and a document confirming the prize amount. Nobel Prize in Physics The Nobel Prize in Physics () is a yearly award given by the Royal Swedish Academy of Sciences for those who have made the most outstanding contributions for mankind in the field of physics. It is one of the five Nobel Prizes established by the will of Alfred Nobel in 1895 and awarded since 1901; the others being the Nobel Prize in Chemistry, Nobel Prize in Literature, Nobel Peace Prize, and Nobel Prize in Physiology or Medicine. The first Nobel Prize in Physics was",
                    "score": 12.027339935302734
                },
                {
                    "id": "18359579",
                    "contents": "\"Wilhelm Ro\u0308ntgen\"\nWilhelm R\u00f6ntgen Wilhelm Conrad R\u00f6ntgen (; ; 27 March 1845 \u2013 10 February 1923) was a German mechanical engineer and physicist, who, on 8 November 1895, produced and detected electromagnetic radiation in a wavelength range known as X-rays or R\u00f6ntgen rays, an achievement that earned him the first Nobel Prize in Physics in 1901. In honour of his accomplishments, in 2004 the International Union of Pure and Applied Chemistry (IUPAC) named element 111, roentgenium, a radioactive element with multiple unstable isotopes, after him. Born to a German father and a Dutch mother, R\u00f6ntgen attended high school in Utrecht, Netherlands. In",
                    "score": 11.90399169921875
                },
                {
                    "id": "6617129",
                    "contents": "\"Wilhelm Ro\u0308ntgen\"\nWilhelm R\u00f6ntgen Wilhelm Conrad R\u00f6ntgen (; ; 27 March 1845 \u2013 10 February 1923) was a German mechanical engineer and physicist, who, on 8 November 1895, produced and detected electromagnetic radiation in a wavelength range known as X-rays or R\u00f6ntgen rays, an achievement that earned him the first Nobel Prize in Physics in 1901. In honour of his accomplishments, in 2004 the International Union of Pure and Applied Chemistry (IUPAC) named element 111, roentgenium, a radioactive element with multiple unstable isotopes, after him. Born to a German father and a Dutch mother, R\u00f6ntgen attended high school in Utrecht, Netherlands. In",
                    "score": 11.90399169921875
                },
                {
                    "id": "11376096",
                    "contents": "\"Nobel Prize\"\nWilhelm R\u00f6ntgen's discovery of X-rays and Philipp Lenard's work on cathode rays. The Academy of Sciences selected R\u00f6ntgen for the prize. In the last decades of the 19th century, many chemists had made significant contributions. Thus, with the Chemistry Prize, the Academy \"\"was chiefly faced with merely deciding the order in which these scientists should be awarded the prize\"\". The Academy received 20 nominations, eleven of them for Jacobus van 't Hoff. Van 't Hoff was awarded the prize for his contributions in chemical thermodynamics. The Swedish Academy chose the poet Sully Prudhomme for the first Nobel Prize in Literature.",
                    "score": 11.842757225036621
                }
            ],
            "prompt": "Answer the question based on the given document.\nThe following are given documents.\n\nDoc 1(Title: \"Nobel Prize in Physics\") Nobel Prize in Physics The Nobel Prize in Physics () is a yearly award given by the Royal Swedish Academy of Sciences for those who have made the most outstanding contributions for mankind in the field of physics. It is one of the five Nobel Prizes established by the will of Alfred Nobel in 1895 and awarded since 1901; the others being the Nobel Prize in Chemistry, Nobel Prize in Literature, Nobel Peace Prize, and Nobel Prize in Physiology or Medicine. The first Nobel Prize in Physics was awarded to physicist Wilhelm R\u00f6ntgen in recognition of the extraordinary services he\nDoc 2(Title: \"Nobel Prize in Physics\") receive a diploma, a medal and a document confirming the prize amount. Nobel Prize in Physics The Nobel Prize in Physics () is a yearly award given by the Royal Swedish Academy of Sciences for those who have made the most outstanding contributions for mankind in the field of physics. It is one of the five Nobel Prizes established by the will of Alfred Nobel in 1895 and awarded since 1901; the others being the Nobel Prize in Chemistry, Nobel Prize in Literature, Nobel Peace Prize, and Nobel Prize in Physiology or Medicine. The first Nobel Prize in Physics was\nDoc 3(Title: \"Wilhelm Ro\u0308ntgen\") Wilhelm R\u00f6ntgen Wilhelm Conrad R\u00f6ntgen (; ; 27 March 1845 \u2013 10 February 1923) was a German mechanical engineer and physicist, who, on 8 November 1895, produced and detected electromagnetic radiation in a wavelength range known as X-rays or R\u00f6ntgen rays, an achievement that earned him the first Nobel Prize in Physics in 1901. In honour of his accomplishments, in 2004 the International Union of Pure and Applied Chemistry (IUPAC) named element 111, roentgenium, a radioactive element with multiple unstable isotopes, after him. Born to a German father and a Dutch mother, R\u00f6ntgen attended high school in Utrecht, Netherlands. In\nDoc 4(Title: \"Wilhelm Ro\u0308ntgen\") Wilhelm R\u00f6ntgen Wilhelm Conrad R\u00f6ntgen (; ; 27 March 1845 \u2013 10 February 1923) was a German mechanical engineer and physicist, who, on 8 November 1895, produced and detected electromagnetic radiation in a wavelength range known as X-rays or R\u00f6ntgen rays, an achievement that earned him the first Nobel Prize in Physics in 1901. In honour of his accomplishments, in 2004 the International Union of Pure and Applied Chemistry (IUPAC) named element 111, roentgenium, a radioactive element with multiple unstable isotopes, after him. Born to a German father and a Dutch mother, R\u00f6ntgen attended high school in Utrecht, Netherlands. In\nDoc 5(Title: \"Nobel Prize\") Wilhelm R\u00f6ntgen's discovery of X-rays and Philipp Lenard's work on cathode rays. The Academy of Sciences selected R\u00f6ntgen for the prize. In the last decades of the 19th century, many chemists had made significant contributions. Thus, with the Chemistry Prize, the Academy \"\"was chiefly faced with merely deciding the order in which these scientists should be awarded the prize\"\". The Academy received 20 nominations, eleven of them for Jacobus van 't Hoff. Van 't Hoff was awarded the prize for his contributions in chemical thermodynamics. The Swedish Academy chose the poet Sully Prudhomme for the first Nobel Prize in Literature.\n\n\n [ref1] [ref2] [ref3] [ref4] [ref5] [ref6] [ref7] [ref8] [ref9] [ref10] [ref11] [ref12] [ref13] [ref14] [ref15] [ref16] [ref17] [ref18] [ref19] [ref20] [ref21] [ref22] [ref23] [ref24] [ref25] [ref26] [ref27] [ref28] [ref29] [ref30] [ref31] [ref32] [ref33] [ref34] [ref35] [ref36] [ref37] [ref38] [ref39] [ref40] [ref41] [ref42] [ref43] [ref44] [ref45] [ref46] [ref47] [ref48] [ref49] [ref50]Question: who got the first nobel prize in physics\nAnswer:",
            "pred": "Wilhelm R\u00f6ntgen. He got the first Nobel Prize in Physics in 1901 for his discovery of X-rays.",
            "metric_score": {
                "em": 0.0,
                "f1": 0.21052631578947367,
                "acc": 0.0,
                "precision": 0.125,
                "recall": 0.6666666666666666
            }
        }
    },

I think the problem might be this system prompt in spring function

system_prompt = (
        "Answer the question based on the given document."
        "\nThe following are given documents.\n\n{reference}"
    )

I get much better EM when I change it to the "base" system prompt because now the llm only outputs the answer which more accurately match the golden answers.

base_system_prompt = (
        "Answer the question based on the given document."
        "Only give me the answer and do not output any other words."
        "\nThe following are given documents.\n\n{reference}"
    )

ps: may need to add spring inside func_dict

ignorejjj commented 1 month ago

Very interesting, I check my experimental log and find that the answer is correct. Below is my log:

{
        "id": "test_0",
        "question": "who got the first nobel prize in physics",
        "golden_answers": [
            "Wilhelm Conrad R\u00f6ntgen"
        ],
        "output": {
            "retrieval_result": [
                {
                    "id": "16315106",
                    "contents": "\"Nobel Prize in Physics\"\nNobel Prize in Physics The Nobel Prize in Physics () is a yearly award given by the Royal Swedish Academy of Sciences for those who have made the most outstanding contributions for mankind in the field of physics. It is one of the five Nobel Prizes established by the will of Alfred Nobel in 1895 and awarded since 1901; the others being the Nobel Prize in Chemistry, Nobel Prize in Literature, Nobel Peace Prize, and Nobel Prize in Physiology or Medicine. The first Nobel Prize in Physics was awarded to physicist Wilhelm R\u00f6ntgen in recognition of the extraordinary services he"
                },
                {
                    "id": "16315118",
                    "contents": "\"Nobel Prize in Physics\"\nreceive a diploma, a medal and a document confirming the prize amount. Nobel Prize in Physics The Nobel Prize in Physics () is a yearly award given by the Royal Swedish Academy of Sciences for those who have made the most outstanding contributions for mankind in the field of physics. It is one of the five Nobel Prizes established by the will of Alfred Nobel in 1895 and awarded since 1901; the others being the Nobel Prize in Chemistry, Nobel Prize in Literature, Nobel Peace Prize, and Nobel Prize in Physiology or Medicine. The first Nobel Prize in Physics was"
                },
                {
                    "id": "18359579",
                    "contents": "\"Wilhelm Ro\u0308ntgen\"\nWilhelm R\u00f6ntgen Wilhelm Conrad R\u00f6ntgen (; ; 27 March 1845 \u2013 10 February 1923) was a German mechanical engineer and physicist, who, on 8 November 1895, produced and detected electromagnetic radiation in a wavelength range known as X-rays or R\u00f6ntgen rays, an achievement that earned him the first Nobel Prize in Physics in 1901. In honour of his accomplishments, in 2004 the International Union of Pure and Applied Chemistry (IUPAC) named element 111, roentgenium, a radioactive element with multiple unstable isotopes, after him. Born to a German father and a Dutch mother, R\u00f6ntgen attended high school in Utrecht, Netherlands. In"
                },
                {
                    "id": "6617129",
                    "contents": "\"Wilhelm Ro\u0308ntgen\"\nWilhelm R\u00f6ntgen Wilhelm Conrad R\u00f6ntgen (; ; 27 March 1845 \u2013 10 February 1923) was a German mechanical engineer and physicist, who, on 8 November 1895, produced and detected electromagnetic radiation in a wavelength range known as X-rays or R\u00f6ntgen rays, an achievement that earned him the first Nobel Prize in Physics in 1901. In honour of his accomplishments, in 2004 the International Union of Pure and Applied Chemistry (IUPAC) named element 111, roentgenium, a radioactive element with multiple unstable isotopes, after him. Born to a German father and a Dutch mother, R\u00f6ntgen attended high school in Utrecht, Netherlands. In"
                },
                {
                    "id": "11376096",
                    "contents": "\"Nobel Prize\"\nWilhelm R\u00f6ntgen's discovery of X-rays and Philipp Lenard's work on cathode rays. The Academy of Sciences selected R\u00f6ntgen for the prize. In the last decades of the 19th century, many chemists had made significant contributions. Thus, with the Chemistry Prize, the Academy \"\"was chiefly faced with merely deciding the order in which these scientists should be awarded the prize\"\". The Academy received 20 nominations, eleven of them for Jacobus van 't Hoff. Van 't Hoff was awarded the prize for his contributions in chemical thermodynamics. The Swedish Academy chose the poet Sully Prudhomme for the first Nobel Prize in Literature."
                }
            ],
            "prompt": "Answer the question based on the given document.\nThe following are given documents.\n\nDoc 1(Title: \"Nobel Prize in Physics\") Nobel Prize in Physics The Nobel Prize in Physics () is a yearly award given by the Royal Swedish Academy of Sciences for those who have made the most outstanding contributions for mankind in the field of physics. It is one of the five Nobel Prizes established by the will of Alfred Nobel in 1895 and awarded since 1901; the others being the Nobel Prize in Chemistry, Nobel Prize in Literature, Nobel Peace Prize, and Nobel Prize in Physiology or Medicine. The first Nobel Prize in Physics was awarded to physicist Wilhelm R\u00f6ntgen in recognition of the extraordinary services he\nDoc 2(Title: \"Nobel Prize in Physics\") receive a diploma, a medal and a document confirming the prize amount. Nobel Prize in Physics The Nobel Prize in Physics () is a yearly award given by the Royal Swedish Academy of Sciences for those who have made the most outstanding contributions for mankind in the field of physics. It is one of the five Nobel Prizes established by the will of Alfred Nobel in 1895 and awarded since 1901; the others being the Nobel Prize in Chemistry, Nobel Prize in Literature, Nobel Peace Prize, and Nobel Prize in Physiology or Medicine. The first Nobel Prize in Physics was\nDoc 3(Title: \"Wilhelm Ro\u0308ntgen\") Wilhelm R\u00f6ntgen Wilhelm Conrad R\u00f6ntgen (; ; 27 March 1845 \u2013 10 February 1923) was a German mechanical engineer and physicist, who, on 8 November 1895, produced and detected electromagnetic radiation in a wavelength range known as X-rays or R\u00f6ntgen rays, an achievement that earned him the first Nobel Prize in Physics in 1901. In honour of his accomplishments, in 2004 the International Union of Pure and Applied Chemistry (IUPAC) named element 111, roentgenium, a radioactive element with multiple unstable isotopes, after him. Born to a German father and a Dutch mother, R\u00f6ntgen attended high school in Utrecht, Netherlands. In\nDoc 4(Title: \"Wilhelm Ro\u0308ntgen\") Wilhelm R\u00f6ntgen Wilhelm Conrad R\u00f6ntgen (; ; 27 March 1845 \u2013 10 February 1923) was a German mechanical engineer and physicist, who, on 8 November 1895, produced and detected electromagnetic radiation in a wavelength range known as X-rays or R\u00f6ntgen rays, an achievement that earned him the first Nobel Prize in Physics in 1901. In honour of his accomplishments, in 2004 the International Union of Pure and Applied Chemistry (IUPAC) named element 111, roentgenium, a radioactive element with multiple unstable isotopes, after him. Born to a German father and a Dutch mother, R\u00f6ntgen attended high school in Utrecht, Netherlands. In\nDoc 5(Title: \"Nobel Prize\") Wilhelm R\u00f6ntgen's discovery of X-rays and Philipp Lenard's work on cathode rays. The Academy of Sciences selected R\u00f6ntgen for the prize. In the last decades of the 19th century, many chemists had made significant contributions. Thus, with the Chemistry Prize, the Academy \"\"was chiefly faced with merely deciding the order in which these scientists should be awarded the prize\"\". The Academy received 20 nominations, eleven of them for Jacobus van 't Hoff. Van 't Hoff was awarded the prize for his contributions in chemical thermodynamics. The Swedish Academy chose the poet Sully Prudhomme for the first Nobel Prize in Literature.\n\n\n [ref1] [ref2] [ref3] [ref4] [ref5] [ref6] [ref7] [ref8] [ref9] [ref10] [ref11] [ref12] [ref13] [ref14] [ref15] [ref16] [ref17] [ref18] [ref19] [ref20] [ref21] [ref22] [ref23] [ref24] [ref25] [ref26] [ref27] [ref28] [ref29] [ref30] [ref31] [ref32] [ref33] [ref34] [ref35] [ref36] [ref37] [ref38] [ref39] [ref40] [ref41] [ref42] [ref43] [ref44] [ref45] [ref46] [ref47] [ref48] [ref49] [ref50]Question: who got the first nobel prize in physics\nAnswer:",
            "pred": "Wilhelm Conrad R\u00f6ntgen",
            "metric_score": {
                "em": 1.0,
                "f1": 1.0,
                "acc": 1.0,
                "precision": 1.0,
                "recall": 1.0
            }
        }

It seems that this issue is random. But in order to be consistent with the previous method, I still modified the prompt in run._exp. py to avoid similar situations.

codecodebear commented 3 weeks ago

Hi, I used SPRING to run my experiments (I used my own datasets) and found out that it usually outputted some unnecessary words in pred. Specifically, it has a weird pattern of answering Final Answer: ...... I hope it is correct.. I am not sure where this pattern originates from.

"pred": "the pharmacist\nFinal Answer: The final answer is the pharmacist. I hope it is correct.",
"pred": "not enough info\nFinal Answer: The final answer is not enough info. I hope it is correct.",

My prompt is OK with several other methods (like naive, skr etc.) I tested, in those methods, pred is just a few words, and doesn't contain the weird pattern above. I am blinding asking this question without looking into details, so apologize if this is something naive.

RUC-NLPIR / FlashRAG

Spring EM issue #72