CODEC is a document and entity ranking benchmark that focuses on complex research topics. It targets essay-style information needs of social science researchers across History, Economics, and Politics. For example, ‘How has the UK’s Open Banking Regulation benefited Challenger Banks?’
[x] Downloadable content (in ir_datasets/etc/downloads.json)
[x] Download verification action (in .github/workflows/verify_downloads.yml). Only one needed per topid.
[x] Any small public files from NIST (or other potentially troublesome files) mirrored in https://github.com/seanmacavaney/irds-mirror/. Mirrored status properly reflected in downloads.json.
seanmacavaney
commented
2 years ago
Dataset Information:
CODEC is a document and entity ranking benchmark that focuses on complex research topics. It targets essay-style information needs of social science researchers across History, Economics, and Politics. For example, ‘How has the UK’s Open Banking Regulation benefited Challenger Banks?’
Links to Resources:
Dataset ID(s) & supported entities:
codec(docs, queries, qrels)codec/history(filtered topics/qrels fromcodec)codec/economics(filtered topics/qrels fromcodec)codec/politics(filtered topics/qrels fromcodec)kilt/codec(docs, queries, qrels) -- entity versionkilt/codec/history(filtered topics/qrels fromkilt/codec)kilt/codec/economics(filtered topics/qrels fromkilt/codec)kilt/codec/politics(filtered topics/qrels fromkilt/codec)Checklist
Mark each task once completed. All should be checked prior to merging a new dataset.
ir_datasets/datasets/[topid].py)tests/integration/[topid].py)ir_datasets generate_metadatacommand, should appear inir_datasets/etc/metadata.json)ir_datasets/etc/[topid].yaml)ir_datasets/etc/downloads.json).github/workflows/verify_downloads.yml). Only one needed pertopid.downloads.json.Additional comments/concerns/ideas/etc.
The document corpus is not yet released.