[WIP] feat: create `c4/en` toy dataset with `entity`, `website_description` and `timestamp` metadata - Githubissues

bigscience-workshop / metadata

Experiments on including metadata such as URLs, timestamps, website descriptions and HTML tags during pretraining.

Apache License 2.0

30 stars 12 forks source link

[WIP] feat: create `c4/en` toy dataset with `entity`, `website_description` and `timestamp` metadata #76

Closed SaulLu closed 2 years ago

SaulLu commented 3 years ago

In this PR you will find the scripts developed for:

extract a sub dataset of c4/en containing 1000 examples randomly extracted from the first 2 files of c4/en (). This dataset is shared on the hub at bs-modeling-metadata/c4-en-reduced.
extract from this dataset the webstite description, timestamp and entity metadata. The resulting dataset is shared on the hub at bs-modeling-metadata/c4-en-reduced-with-metadata

Some investigations have to be done on the resulting toy dataset before merging this PR