Refactor a util function

bigscience-workshop / metadata

Experiments on including metadata such as URLs, timestamps, website descriptions and HTML tags during pretraining.

Apache License 2.0

30 stars 12 forks source link

Closed cccntu closed 2 years ago

cccntu commented 2 years ago

This can be reused in evaluation script