ClueWeb22

[janheinrichmerker]

Dataset Information:

ClueWeb22 is the newest in the Lemur Project's ClueWeb line of datasets that support research on information retrieval, natural language processing and related human language technologies. This new dataset is being developed by the Lemur Project with significant assistance and support from Microsoft Corporation.

The ClueWeb22 dataset has several novel characteristics compared with earlier ClueWeb datasets.

• It is much larger.
• Documents are of higher quality.
• Documents are provided in several formats (HTML, clean text, screen shots).
• Document page analyses are provided that reveal where on a page text was displayed, and what was near it.
• The dataset includes a large set of crowdsourced queries and shallow relevance assessments (a pseudo search log).

Authors: Arnold Overwijk, Chenyan Xiong (@xiongchenyan), Jamie Callan (@jamiecallan), Cameron VandenBerg, Xiao Lucy Liu

Links to Resources:

• Website: https://lemurproject.org/clueweb22/index.php
• Documents spec: https://lemurproject.org/clueweb22/docspecs.php
• Queries spec: https://lemurproject.org/clueweb22/qryspecs.php
• SIRIP paper: https://doi.org/10.1145/3477495.3536321

Dataset ID(s) & supported entities:

• clueweb22/a: 200M docs, queries, qrels, scoreddocs?
• clueweb22/b: 2B docs, queries?, qrels?, scoreddocs?
• clueweb22/l: 10B docs, queries?, qrels?, scoreddocs?

Checklist

Mark each task once completed. All should be checked prior to merging a new dataset.

• [x] Dataset definition (in ir_datasets/datasets/clueweb22.py)
• [x] Tests (in tests/integration/clueweb22.py)
• [x] Metadata generated (using ir_datasets generate_metadata command, should appear in ir_datasets/etc/metadata.json)
• [x] Documentation (in ir_datasets/etc/clueweb22.yaml)
  • [ ] Documentation generated in https://github.com/seanmacavaney/ir-datasets.com/
• [x] Downloadable content (in ir_datasets/etc/downloads.json) Manual download requirded.
  • [x] Download instructions added
  • [x] Download verification action (in .github/workflows/verify_downloads.yml). Only one needed per topid.
  • [x] Any small public files from NIST (or other potentially troublesome files) mirrored in https://github.com/seanmacavaney/irds-mirror/. Mirrored status properly reflected in downloads.json.

Additional comments/concerns/ideas/etc.

The dataset is planned to be used for shared tasks in the near future. I also personally think it is of very high value to have this in ir_datasets.

Open Questions

• [x] Where to get the topic tag mentioned in the paper?
• [x] Is VDOM-Paragraph the same as VDOM-Passage in the WARC headers?
• [x] What means the ? in the inlink format anchor type description?

[seanmacavaney]

Excellent, thanks @heinrichreimer!

A while back I requested that they include offset files to facilitate random lookups, and it looks like it made it into the final spec! This will make adding the datasets much easier, since we won't need to save zlib states and release our own checkpoint files.

[jamiecallan]

Sean is correct.  Each warc.gz file is compressed by record and has a companion offset file.  To get the HTML of a specific document, open the appropriate .warc.offset file (can be determined from the ClueWeb docid), fseek to find the byte offsets of the start/end of the document (also determined from the ClueWeb docid), open the .warc.gz file, fseek to the start of the document, read the bytes, and uncompress them.

We can provide data samples if you need them.

We are trying to apply this or a similar architecture to all other types of data in the dataset, so that everything can be accessed quickly given a ClueWeb docid.

Best,

    Jamie

On 10/5/2022 7:09 AM, Sean MacAvaney wrote:

Excellent, thanks @heinrichreimer https://github.com/heinrichreimer!

A while back I requested that they include offset files to facilitate random lookups, and it looks like it made it into the final spec! This will make adding the datasets much easier, since we won't need to save zlib states and release our own checkpoint files.

— Reply to this email directly, view it on GitHub https://github.com/allenai/ir_datasets/issues/210#issuecomment-1268287629, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRMTZKVX77ZWOQ2LFPDT6TWBVOWHANCNFSM6AAAAAAQ5O3JSY. You are receiving this because you were mentioned.Message ID: @.***>

[janheinrichmerker]

Thanks for explaining the file structure! A small data sample would indeed be nice. Then we can start implementing the required parsers for ir_datasets even before our full copy arrives. @seanmacavaney do you want to do it or should I take some time to implement the parser then?

[seanmacavaney]

We're still in the process of requesting the data here. A sample would indeed be helpful for getting started.

@heinrichreimer -- I've got a pretty busy couple of weeks coming up, would you be able to take a stab at the implementation?

[janheinrichmerker]

Sure, I'll try my best. I guess most of the code can be "recycled" from ClueWeb12 anyway.

[seanmacavaney]

Awesome, thanks! The most challenging bit is doing lookups, but with the offset file that's included, this should be much easier.

Feel free to reach out if you have problems/questions/etc. Thanks!

[janheinrichmerker]

As ClueWeb22 also features language tags and is structured in a way to efficiently filter by language, I'll also include subsets like this:

• clueweb22/a/en
• clueweb22/a/de
• clueweb22/a/zn
• ...
• clueweb22/a/other-languages
• clueweb22/b/en
• ...
• clueweb22/l/en
• ...

[seanmacavaney]

Great, thanks. This is aligned with clueweb09/[lang]

[janheinrichmerker]

As the categories are subsets of the larger ones, I've now also added "views" that can, for example, be used to just parse the plain text from the B category. The keys would be clueweb22/b/as-l, clueweb22/b/as-a, clueweb22/b/as-l/en, clueweb22/b/as-a/en and so on. To not clutter the list of dataset IDs too much, we could also just skip the language-specific versions for the "views".