While reviewing the data in huggingface hub, I observed duplicate annotations in annotations.json, for example, the annotation below has repeat 3 times in annotations.json:
{
"prompt": "Explain Ab Klink, including information about party, birth place",
"output": "Ab Klink is a Dutch politician who was born on 2 November 1958 in Stellendam, Netherlands. He is a member of the Christian Democratic Appeal Party (CDA) and served as Minister of Health, Welfare and Sport from 2007 to 2010. In addition to his political career, Klink worked as a professor at the Vrije Universiteit Amsterdam, focusing on healthcare-related research.",
"annotated": "Ab Klink is a Dutch politician who was born on 2 November 1958 in Stellendam, Netherlands. He is a member of the Christian Democratic Appeal Party (CDA) and served as Minister of Health, Welfare and Sport from 2007 to 2010. In addition to his political career, Klink worked as a professor at the Vrije Universiteit Amsterdam, focusing on healthcare-related research.",
"subject": "ab klink",
"dataset": "webnlg",
"model": "chatgpt"
},
And some annotations seem to be incorrect and contradictory, as shown below:
{
"prompt": "Explain Ab Klink, including information about party, birth place",
"output": "Ab Klink is a well-known Dutch politician who was born on September 2, 1958, in Groningen, Netherlands. He is a member of the Christian Democratic Appeal (CDA) party, which is a center-right political party in the Netherlands.Klink studied law at the University of Groningen and later worked as a lawyer before entering politics. He was first elected to the House of Representatives in 1989 and has served in various positions within the CDA party, including as party chairman from 1997 to 2001.In 2001, Klink was appointed as Minister of Agriculture, Nature and Food Quality in the Cabinet Balkenende I, a position he held until 2003. He then served as Minister of Defense in the Cabinet Balkenende II from 2003 to 2007.In 2010, Klink was elected as a Member of the European Parliament, where he served on the Committee on Agriculture and Rural Development. He was re-elected in 2014 and served until 2019.Klink is known for his advocacy for Christian democratic values and his commitment to issues related to agriculture, nature conservation, and defense. He is also a strong supporter of European integration and has played a key role in shaping the EU's agricultural and rural development policies.In addition to his political career, Klink has also served on various boards and committees, including as chairman of the Netherlands' Advisory Council for Science and Technology and as a member of the European Academy of Sciences and Arts.Overall, Ab Klink is a respected and experienced politician who has made significant contributions to Dutch and European politics, particularly in the areas of agriculture, defense, and Christian democratic values.",
"annotated": " Ab Klink is a well-known Dutch politician who was born on September 2, 1958, in <entity><delete>Groninge</delete><mark>stellendam</mark></entity>n, Netherlands. He is a member of the Christian Democratic Appeal (CDA) party, which is a center-right political party in the Netherlands.<contradictory><delete>Klink studied law at the University of Groningen and later worked as a lawyer before entering politics.</delete></contradictory> He was first elected to the House of Representatives in <entity><delete>1989</delete><mark>2010</mark></entity> and has served in various positions within the CDA party<contradictory><delete>, including as party chairman from 1997 to 2001.</delete></contradictory><contradictory><delete>In 2001, Klink was appointed as Minister of Agriculture, Nature and Food Quality in the Cabinet Balkenende I, a position he held until 2003</delete></contradictory>. <contradictory><delete>He then served as Minister of Defense in the Cabinet Balkenende II from 2003 to 2007.</delete></contradictory><contradictory><delete>In 2010, Klink was elected as a Member of the European Parliament, where he served on the Committee on Agriculture and Rural Development. </delete></contradictory><contradictory><delete>He was re-elected in 2014 and served until 2019.</delete></contradictory>Klink is known for his advocacy for Christian democratic values <contradictory><delete>and his commitment to issues related to agriculture, nature conservation, and defense.</delete></contradictory> <unverifiable><delete>He is also a strong supporter of European integration and has played a key role in shaping the EU's agricultural and rural development policies.</delete></unverifiable><contradictory><delete>In addition to his political career, Klink has also served on various boards and committees, including as chairman of the Netherlands' Advisory Council for Science and Technology and as a member of the European Academy of Sciences and Arts.</delete></contradictory>Overall, Ab Klink is a respected and experienced politician who has made significant contributions to Dutch and European politics, particularly in the areas of <contradictory><delete>agriculture, defense, and</delete></contradictory> Christian democratic values.",
"subject": "ab klink",
"dataset": "webnlg",
"model": "llama"
},
and
{
"prompt": "Explain Ab Klink, including information about party, birth place",
"output": * SAME AS ABOVE *,
"annotated": " Ab Klink is a well-known Dutch politician who was born on <entity><delete>Septembe</delete><mark>november</mark></entity>r 2, 1958, in Groninge<entity><delete>n, Netherl</delete><mark>stellendam</mark></entity>ands. He is a member of the Christian Democratic Appeal (CDA) party, which is a center-right political party in the Netherlands.<invented><delete>Klink studied law at the University of Groningen and later worked as a lawyer before entering politics</delete></invented>. He was first elected to the House of Representatives in <entity><delete>1989</delete><mark>2010</mark></entity> and has served in various positions within the CDA party, including as party chairman from 1997 to 2001.<unverifiable><delete>In 2001, Klink was appointed as Minister of Agriculture, Nature and Food Quality in the Cabinet Bal</delete></unverifiable>kenende I, a position he held until 2003. He then served as Minister of Defense in the Cabinet Balkenende I<entity><delete>I f</delete><mark>4</mark></entity>rom 2003 to 2007<entity><delete>.In 2010, K</delete><mark>2007-2010</mark></entity>lin<invented><delete>k was elected as a Member of the European Parliament, where he served on the Committee on Agriculture and Rural Development. He was re-elected in 2014 and served until 2019.</delete></invented>Klink is known for his advocacy for Christian democratic values and his commitment to issues related to agriculture, nature conservation, and defense. He is also a strong supporter of European integration and has played a key role in shaping the EU's agricultural and rural development policies.In addition to his political career, Klink has also served on various boards and committees, including as chairman of the Netherlands' Advisory Council for Science and Technology and as a member of the European Academy of Sciences and Arts.Overall, Ab Klink is a respected and experienced politician who has made significant contributions to Dutch and European politics, particularly in the areas of agriculture, defense, and Christian democratic values.",
"subject": "ab klink",
"dataset": "webnlg",
"model": "llama"
},
Contradictory of annotation: For the same output, the first annotation labels many \<contradictory> but none \<invented>, and the last annotation labels many \<invented> but none \<contradictory>. In the detection task, which hallucination type should we label for this output? It's \<contradictory> or \<invented> or both?
Wrong annotation: In the first annotation, the incorrect date of birth was not marked; and the word is cut incorrectly in Ab Klink... was born on <entity><delete>Septembe</delete><mark>november</mark></entity>r 2, 1958, in Groninge<entity><delete>n, Netherl</delete><mark>stellendam</mark></entity>ands.
Thanks for your brilliant work!
While reviewing the data in huggingface hub, I observed duplicate annotations in annotations.json, for example, the annotation below has repeat 3 times in annotations.json:
And some annotations seem to be incorrect and contradictory, as shown below:
and
Ab Klink... was born on <entity><delete>Septembe</delete><mark>november</mark></entity>r 2, 1958, in Groninge<entity><delete>n, Netherl</delete><mark>stellendam</mark></entity>ands.