bigscience-workshop / metadata

Experiments on including metadata such as URLs, timestamps, website descriptions and HTML tags during pretraining.
Apache License 2.0
30 stars 12 forks source link

feat: find a solution to load the dataset #122

Open SaulLu opened 2 years ago

SaulLu commented 2 years ago

While testing the real data extraction, I encountered a new problem: the websites descriptions are rarely present in the metadata_website_desc column. Therefore, the datasets library cannot load such a dataset by having to guessing the feature type. It has to know them beforehand.