CLEF eHealth ad-hoc search

[seanmacavaney]

The CLEF eHealth dataset appears to be publicly available (common crawl)

• 2018: https://clefehealth.imag.fr/?page_id=330
• (no consumer health search track in 2019?)
• 2020: https://clefehealth.imag.fr/?page_id=189
• 2021: https://clefehealth.imag.fr/?page_id=610 (no info at time of writing)

[seanmacavaney]

CLEF eHealth 2016-17 is handled now (it used the CW12b13 collection).

2018 onwards has a new document collection that provides some challenges.

• It's hosted on Dropbox, which has caused me problems in the past for automatic downloads. (Or maybe that was Box?) I guess worst case we can ask the user to manually download and move into place.
• The file is a massive .tar.gz, where each document is its own file. It's organized like: [domain]/[guid].
• It's 96GB compressed, 480 uncomrpessed. So it probably falls under the "too large build docstore" category. Luckily, the IDs under each domain are sorted. So we should be able to use the zlib-state approach to build a checkpoint file for lookups. I'm not sure how well it will play with python's tarfile, but in theory it should work.

[seanmacavaney]

Correction: Box is the annoying one. Dropbox is easy, you just need to set dl=1. So this is a direct download link for the collection: https://www.dropbox.com/s/ixnqt33u5xeelth/clef2018collection.tar.gz?dl=1