CLEF Touché

[seanmacavaney]

Dataset Information:

For argument retrieval (Task 1)

Links to Resources:

• https://webis.de/events/touche-20/
• https://link.springer.com/chapter/10.1007/978-3-030-58219-7_26
• https://link.springer.com/chapter/10.1007/978-3-030-85251-1_28

Dataset ID(s):

• clef-touche/v1 (docs)
• clef-touche/v1/task1-2020 (docs, queries, qrels)
• clef-touche/2020-04-01 (docs)
• clef-touche/2020-04-01/task1-2020 (docs, queries, qrels) -- note: this is a distinct set of labels from clef-touche/v1/task1-2020 -- refer to email "touché 2020 task 1 qrels" for details
• clef-touche/2020-04-01/task1-2021 (docs, queries, qrels) -- includes both quality and relevance judgments

Supported Entities

• [x] docs
• [x] queries
• [x] qrels
• [ ] scoreddocs
• [ ] docpairs

Additional comments/concerns/ideas/etc.

Refer to email "touché 2020 task 1 qrels" for more details about the various versions of the qrels at play here. Specifically, there are separate qrels used for beir/webis-touche -- in the old version, a couple of qrels were missing, and the new version maps the revised 2020-04-01/task1-2020 qrels onto the v1 corpus. IRDS seems like a good place to document all this.

[seanmacavaney]

@heinrichreimer - here's the related issue

[seanmacavaney]

@heinrichreimer -- interested in adding the touche queries/qrels in too?

[janheinrichmerker]

Sure. I'm going to talk to Sascha directly regarding the different datasets.

[seanmacavaney]

Awesome, thanks! I'm keen to get these added, so let me know if I can be of assistance.