search

bigai-nlco / LooGLE

ACL 2024 | LooGLE: Long Context Evaluation for Long-Context Language Models
MIT License
147 stars 6 forks source link

It seems that the annotated original text are not provided? #11

Open waylonli opened 3 hours ago

waylonli commented 3 hours ago

For long dependency QA, I suppose the 'S' key under 'qa_pairs' should be the annotated original text? But I found that most of the time the context in 'S' cannot be found in 'input'. Am I misunderstanding what 'S' is?

waylonli commented 3 hours ago

For example: loogle_data[1]['qa_pairs'][0]['S'][0] is

'Ciutat Vella:\n[Population (2015): 100,685;\nArea: 4.49 km2;\nDensity = 100,685 / 4.49 = 22,431 people/km2.]\nEixample:\n[Population: 263,565;\nArea: 7.46 km2;\nDensity = 263,565 / 7.46 = 35,338 people/km2.]\nSants-Montju?c:\n[Population: 180,824;\nArea: 21.35 km2;\nDensity = 180,824 / 21.35 = 8,473 people/km2.]\nLes Corts:\n[Population: 81,200;\nArea: 6.08 km2;\nDensity = 81,200 / 6.08 = 13,355 people/km2.]\nSarrià-Sant Gervasi:\n[Population: 145,761;\nArea: 20.09 km2;\nDensity = 145,761 / 20.09 = 7,254 people/km2.]\nGràcia:\n[Population: 120,273;\nArea: 4.19 km2;\nDensity = 120,273 / 4.19 = 28,710 people/km2.]\nHorta-Guinardó:\n[Population: 166,950;\nArea: 11.96 km2;\nDensity = 166,950 / 11.96 = 13,959 people/km2.]\nNou Barris:\n[Population: 164,516;\nArea: 8.04 km2;\nDensity = 164,516 / 8.04 = 20,466 people/km2.]\nSant Andreu:\n[Population: 145,983;\nArea: 6.56 km2;\nDensity = 145,983 / 6.56 = 22,246 people/km2.]\nSant Martí:\n[Population: 232,629;\nArea: 10.80 km2;\nDensity = 232,629 / 10.80 = 21,540 people/km2.]'

I completely cannot find such paragraph or sentences in the original input.