CODEC is a document and entity ranking benchmark that focuses on complex research topics. It targets essay-style information needs of social science researchers across History, Economics, and Politics. For example, ‘How has the UK’s Open Banking Regulation benefited Challenger Banks?’
[x] Downloadable content (in ir_datasets/etc/downloads.json)
[x] Download verification action (in .github/workflows/verify_downloads.yml). Only one needed per topid.
[x] Any small public files from NIST (or other potentially troublesome files) mirrored in https://github.com/seanmacavaney/irds-mirror/. Mirrored status properly reflected in downloads.json.
Dataset Information:
CODEC is a document and entity ranking benchmark that focuses on complex research topics. It targets essay-style information needs of social science researchers across History, Economics, and Politics. For example, ‘How has the UK’s Open Banking Regulation benefited Challenger Banks?’
Links to Resources:
Dataset ID(s) & supported entities:
codec
(docs, queries, qrels)codec/history
(filtered topics/qrels fromcodec
)codec/economics
(filtered topics/qrels fromcodec
)codec/politics
(filtered topics/qrels fromcodec
)kilt/codec
(docs, queries, qrels) -- entity versionkilt/codec/history
(filtered topics/qrels fromkilt/codec
)kilt/codec/economics
(filtered topics/qrels fromkilt/codec
)kilt/codec/politics
(filtered topics/qrels fromkilt/codec
)Checklist
Mark each task once completed. All should be checked prior to merging a new dataset.
ir_datasets/datasets/[topid].py
)tests/integration/[topid].py
)ir_datasets generate_metadata
command, should appear inir_datasets/etc/metadata.json
)ir_datasets/etc/[topid].yaml
)ir_datasets/etc/downloads.json
).github/workflows/verify_downloads.yml
). Only one needed pertopid
.downloads.json
.Additional comments/concerns/ideas/etc.
The document corpus is not yet released.