[QA] DPR 오픈소스를 쓰기 위한 데이터 형식 매칭

[ebbunnim]

질문

• 새로운 Dense Retrieval를 고민하다가 DPR 오픈소스를 사용해보려고 합니다.
• 그런데, training 시에 기존에 학습된 데이터셋 형태로 맞춰달라는 요구사항이 있었습니다.

  
  {
      "dataset": str,
      "question": str,
      "answers": list of str
      "positive_ctxs": list of dictionaries of format {'title': str, 'text': str, 'score': int, 'title_score': int, 'passage_id': str}
      "negative_ctxs": list of dictionaries of format {'title': str, 'text': str, 'score': int, 'title_score': int, 'passage_id': str}
      "hard_negative_ctxs": list of dictionaries of format {'title': str, 'text': str, 'score': int, 'title_score': int, 'passage_id': str}
  }

[출처](https://github.com/deepset-ai/haystack/blob/master/tutorials/Tutorial9_DPR_training.ipynb)

`negative_ctxs`은 저희가 논의한 negative sample을 넣는 컬럼 같습니다!
조교님 추천에 계속 등장하는 논문의 구현체들이라, 꼭 사용해보고 싶은데 늘 데이터셋 형식을 맞추는 것에서 발목을 잡히네요.

혹시 이 오픈소스를 사용할 수 있는 방법 (haystack은 DPR 오픈소스를 쉽게 사용할 수 있도록 도와주는 오픈소스입니다) 떠오르시는 분들은 알려주시면 좋겠습니다. ㅠㅠ 

[ggm1207]

make_dataset에 있는 코드에서 코드만 살짝 수정하면 비슷한 데이터셋으로 만들 수 있을 것 같아요! 이따가 같이 해보시죠

[ebbunnim]

엇 네네 좋습니다!

[ebbunnim]

그런데 저 score랑 negative sample은 저희가 임의로 만드는 걸로 생각하고 계시나요? score가 제일 문제라고 생각해서요 ㅠㅠ 유사 정도를 저희가 추정할 수 있을까? 이런 의구심이 있습니다..

[ggm1207]

score하고 title_score가 뭘 의미하는 지 알 수 있을까요? 저 데이터셋 참고해서 의미를 알면 맞출 수 있을 것 같습니다. score하고 title_score 타입이 int네요..?

[ebbunnim]

지금 이슈 게시판을 엄청 검색해봤는데, score은 bm25에서 나온 스코어인 것 같습니다! title score는 아직 못찾았어요 ㅠㅠ 더 찾아볼게요

https://github.com/facebookresearch/DPR/issues/91, https://github.com/facebookresearch/DPR/issues/20

[ggm1207]

넵넵, 역시 Human Retrieval ... 어떻게 이렇게 빨리

[ebbunnim]

negative_ctxs is not actually used in Haystack's DPR training so we recommend you set it to an empty list. They were used by the original DPR authors in an experiment to compare it against the in-batch negatives method.

• hard도 empty 가능한 것 같아서 이따가 만들어보면서 비어도 되는지 확인해봐야 할 것 같습니다!

[ebbunnim]

[
    {
        "dataset": "squad1-train-psgs_w100",
        "question": "To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?",
        "answers": [
            "Saint Bernadette Soubirous"
        ],
        "positive_ctxs": [
            {
                "title": "The Song of Bernadette (novel)",
                "text": "The Song of Bernadette (novel) The Song of Bernadette (German: \"Das Lied von Bernadette\") is a 1941 novel that tells the story of Saint Bernadette Soubirous, who, from February to July 1858 reported eighteen visions of the Blessed Virgin Mary in Lourdes, France. The novel was written by Franz Werfel and translated into English by Lewis Lewisohn in 1942. It was extremely popular, spending more than a year on the \"New York Times\" Best Seller list and 13 weeks in first place. The novel was adapted into the 1943 film \"The Song of Bernadette\" starring Jennifer Jones. Franz Werfel was",
                "score": 15.853051,
                "title_score": 0,
                "psg_id": "7427364"
            },
            {
                "title": "Basilica of St. Pius X",
                "text": "Basilica of St. Pius X The Basilica of Saint Pius X, informally known as the Underground Basilica, is a large Roman Catholic church and minor basilica, located in the town of Lourdes, France. It is part of the Sanctuary of Our Lady of Lourdes. Lourdes is a major Catholic pilgrimage site and the Catholic Church endorses the belief that the Virgin Mary appeared to Saint Bernadette Soubirous there. The Basilica of St. Pius X is the largest and most controversial of the Domain's churches. It was completed in 1958 in anticipation of the enormous crowds expected in Lourdes for the",
                "score": 15.052247,
                "title_score": 0,
                "psg_id": "9910383"
            }
        ],
        "negative_ctxs": [
            {
                "title": "Cormac McCarthy",
                "text": "chores of the house, Lee was asked by Cormac to also get a day job so he could focus on his novel writing. Dismayed with the situation, she moved to Wyoming, where she filed for divorce and landed her first job teaching. Cormac McCarthy is fluent in Spanish and lived in Ibiza, Spain, in the 1960s and later settled in El Paso, Texas, where he lived for nearly 20 years. In an interview with Richard B. Woodward from \"The New York Times\", \"McCarthy doesn't drink anymore \u2013 he quit 16 years ago in El Paso, with one of his young",
                "score": 0,
                "title_score": 0,
                "psg_id": "2145653"
            },
            {
                "title": "Pragmatic Sanction of 1549",
                "text": "one heir, Charles effectively united the Netherlands as one entity. After Charles' abdication in 1555, the Seventeen Provinces passed to his son, Philip II of Spain. The Pragmatic Sanction is said to be one example of the Habsburg contest with particularism that contributed to the Dutch Revolt. Each of the provinces had its own laws, customs and political practices. The new policy, imposed from the outside, angered many inhabitants, who viewed their provinces as distinct entities. It and other monarchical acts, such as the creation of bishoprics and promulgation of laws against heresy, stoked resentments, which fired the eruption of",
                "score": 0,
                "title_score": 0,
                "psg_id": "2271902"
            },
            {
                "title": "Hall Airport",
                "text": "Hall Airport Hall Airport is a privately owned, public use airport located six nautical miles (11 km) northwest of the central business district of Kaufman, a city in Kaufman County, Texas, United States. Hall Airport covers an area of 27 acres (11 ha) at an elevation of 440 feet (134 m) above mean sea level. It has one runway designated 17/35 with a turf surface measuring 2,585 by 40 feet (788 x 12 m). For the 12-month period ending May 23, 2007, the airport had 201 general aviation aircraft operations, an average of 16 per month. At that time there",
                "score": 0,
                "title_score": 0,
                "psg_id": "17333840"
            }
        ],
        "hard_negative_ctxs": [
            {
                "title": "Lourdes apparitions",
                "text": "to Soubirous, her visions occurred at the grotto of Massabielle, just outside Lourdes. On 16 July 1858, Soubirous visited the grotto for the last time and said: \"I have never seen her so beautiful before.\" On 18 January 1860, the local bishop declared: \"The Virgin Mary did appear indeed to Bernadette Soubirous.\" In 1958, Pope Pius XII issued the encyclical \"Le p\u00e8lerinage de Lourdes\" (\"The pilgrimage to Lourdes\") on the 100th anniversary of the apparitions. Pope John Paul II visited Lourdes three times; Pope Benedict XVI visited Lourdes on 15 September 2008 to commemorate the 150th anniversary of the apparitions.",
                "score": 21.230358,
                "title_score": 0,
                "psg_id": "10392708"
            },
            {
                "title": "Our Lady of Lourdes",
                "text": "doctor stated that Soubirous, in ecstasy, was observed to have held her hands over a lit candle without sustaining harm. On 16 July, Soubirous went for the last time to the Grotto. \"I have never seen her so beautiful before,\" she reported. The Church, faced with nationwide questions, decided to institute an investigative commission on 17 November 1858. On 18 January 1860, the local bishop finally declared that: \"The Virgin Mary did appear indeed to Bernadette Soubirous.\" These events established the Marian veneration in Lourdes, which together with F\u00e1tima, is one of the most frequented Marian shrines in the world,",
                "score": 19.483299,
                "title_score": 0,
                "psg_id": "6948016"
            },
            {
                "title": "Lourdes water",
                "text": "Lourdes water Lourdes water is water which flows from a spring in the Grotto of Massabielle in the Sanctuary of Our Lady of Lourdes, France. The location of the spring was described to Bernadette Soubirous by an apparition of Our Lady of Lourdes on 25 February 1858. Since that time many thousands of pilgrims to Lourdes have followed the instruction of Our Lady of Lourdes to \"drink at the spring and bathe in it\". Lourdes water is considered non-liturgical holy water. Although never formally encouraged by the Church, Lourdes water has become a focus of devotion to the Virgin Mary",
                "score": 19.409706,
                "title_score": 0,
                "psg_id": "11782434"
            },
            {
                "title": "Our Lady of Lourdes",
                "text": "Our Lady of Lourdes Our Lady of Lourdes is a Roman Catholic title of the Blessed Virgin Mary venerated in honour of the Marian apparitions that reportedly occurred in 1858 in the vicinity of Lourdes in France. The first of these is the apparition of 11 February 1858, when 14-year old Bernadette Soubirous told her mother that a \"lady\" spoke to her in the cave of Massabielle (a kilometre and a half (1 mi) from the town) while she was gathering firewood with her sister and a friend. Similar apparitions of the \"Lady\" were reported on seventeen occasions that year,",
                "score": 19.405451,
                "title_score": 0,
                "psg_id": "6948036"
            },
            {
                "title": "The Song of Bernadette (film)",
                "text": "\"The Song of Bernadette\" was presented on \"Hollywood Star Time\" April 21, 1946. The 30-minute adaptation starred Vincent Price, Lee J. Cobb, Pedro DeCordoba, and Vanessa Brown. The Song of Bernadette (film) The Song of Bernadette is a 1943 biographical drama film based on the 1941 novel of the same name by Franz Werfel. It stars Jennifer Jones in the title role, which tells the story of Bernadette Soubirous (later canonized Saint Bernadette) who, from February to July 1858 in Lourdes, France, reported eighteen visions of the Blessed Virgin Mary. The film was directed by Henry King, from a screenplay",
                "score": 15.840563,
                "title_score": 0,
                "psg_id": "834915"
            },
        ]
    },
-- 다른 데이터셋 반복 
]

[ebbunnim]

https://huggingface.co/transformers/model_doc/dpr.html

[ebbunnim]

<입출력 관련 클래스 - Document 클래스를 list로 묶어서 반환>

<dense retrieval 구현체 위치> https://github.com/deepset-ai/haystack/blob/a06e4450d1f039a48d06ca6b9345d50fbc0fbf79/haystack/retriever/dense.py