koustuvsinha
commented
1 year ago Hi, yes unfortunately this is a known issue with several annotations. We have now provided a sanitized version of the annotation in the develop branch, and documented it here : https://koustuvsinha.com/post/zero_shot_clutrr/. Hope this helps!
hi i found
data_06b8f2a11.3_test.csvappears to include lots of wrong annotations. for example,{"Unød: 0": 2, "id": "fe81eae5-c860-417f-8272-fbea0585d016", "story": "[Kathleen] was excited because she was meeting her father, [Henry], for lunch. [Howard] and his son [Wayne] went to look at cars. [Howard] ended up buying the Mustang. [Howard] likes to spend time with his aunt, [Kathleen], who was excellent at cooking chicken.", "query": ["Wayne", "Henry"], "text_query": NaN, "target": "father", "text_target": ["[Henry] was so proud of his son, [Wayne]. he received a great scholarship to college."], "clean_story": "[Howard] and his son [Wayne] went to look at cars. [Howard] ended up buying the Mustang. [Howard] likes to spend time with his aunt, [Kathleen], who was excellent at cooking chicken. [Kathleen] was excited because she was meeting her father, [Henry], for lunch.", "proof_state": [{"('Wayne', 'father', 'Henry')": [["Wayne", "sister", "Kathleen"], ["Kathleen", "father", "Henry"]]}, {"('Wayne', 'sister', 'Kathleen')": [["Wayne", "son", "Howard"], ["Howard", "aunt", "Kathleen"]]}], "f_comb": "son-aunt-father", "task_name": "task_1.3", "story_edges": [[0, 1], [1, 2], [2, 3]], "edge_types": ["son", "aunt", "father"], "query_edge": [0, 3], "genders": "Wayne:male,Howard:male,Kathleen:female,Henry:male", "syn_storøy": NaN, "node_mapping": {"16": 0, "17": 1, "3": 2, "0": 3}, "task_split": "test"}Henry is definitely NOT Wayne's father?! In general, lots of mother/father-in-laws were incorrectly annotated as father/mother. Did I miss anything here?