Add a metadata field for when source data was produced

[davanstrien]

Is your feature request related to a problem? Please describe. The current problem is that information about when source data was produced is not easily visible. Though there are a variety of metadata fields available in the dataset viewer, time period information is not included. This feature request suggests making metadata relating to the time that the underlying source data was produced more prominent and outlines why this specific information is of particular importance, both in domain-specific historic research and more broadly.

Describe the solution you'd like

There are a variety of metadata fields exposed in the dataset viewer (license, task categories, etc.) These fields make this metadata more prominent both for human users and as potentially machine-actionable information (for example, through the API). I would propose to add a metadata field that says when some underlying data was produced. For example, a dataset would be labelled as being produced between 1800-1900.

Describe alternatives you've considered This information is sometimes available in the Datacard or a paper describing the dataset. However, it's often not that easy to identify or extract this information, particularly if you want to use this field as a filter to identify relevant datasets.

Additional context

I believe this feature is relevant for a number of reasons:

• Increasingly, there is an interest in using historical data for training language models (for example, https://huggingface.co/dbmdz/bert-base-historic-dutch-cased), and datasets to support this task (for example, https://huggingface.co/datasets/bnl_newspapers). For these datasets, indicating the time periods covered is particularly relevant.
• More broadly, time is likely a common source of domain drift. Datasets of movie reviews from the 90s may not work well for recent movie reviews. As the documentation and long-term management of ML data become more of a priority, quickly understanding the time when the underlying text (or other data types) is arguably more important.
• time-series data: datasets are adding more support for time series data. Again, the periods covered might be particularly relevant here.

open questions

• I think some of my points above apply not only to the underlying data but also to annotations. As a result, there could also be an argument for encoding this information somewhere. However, I would argue (but could be persuaded otherwise) that this is probably less important for filtering. This type of context is already addressed in the datasheets template and often requires more narrative to discuss.
• what level of granularity would make sense for this? e.g. assigning a decade, century or year?
• how to encode this information? What formatting makes sense
• what specific time to encode; a data range? (mean, modal, min, max value?)

This is a slightly amorphous feature request - I would be happy to discuss further/try and propose a more concrete solution if this seems like something that could be worth considering. I realise this might also touch on other parts of the 🤗 hubs ecosystem.

[severo]

A question to the datasets maintainers: is there a policy about how the set of allowed metadata fields is maintained and expanded?

Metadata are very important, but defining the standard is always a struggle between allowing exhaustivity without being too complex. Archivists have Dublin Core, open data has https://frictionlessdata.io/, geo has ISO 19139 and INSPIRE, etc. and it's always a mess! I'm not sure we want to dig too much into it, but I'm curious to know if there has been some work on the metadata standard.

[davanstrien]

Metadata are very important, but defining the standard is always a struggle between allowing exhaustivity without being too complex. Archivists have Dublin Core, open data has frictionlessdata.io, geo has ISO 19139 and INSPIRE, etc. and it's always a mess! I'm not sure we want to dig too much into it, but I'm curious to know if there has been some work on the metadata standard.

I thought this is a potential issue with adding this field since it might be hard to define what is general enough to be useful for most data vs what becomes very domain-specific. Potentially adding one extra field leads to more and more fields in the future.

Another issue is that there are some metadata standards around data i.e. datacite, but not many aimed explicitly at ML data afaik. Some of the discussions around metadata for ML are also more focused on versioning/managing data in production environments. My thinking is that here, some reference to the time of production would also often be tracked/relevant, i.e. for triggering model training, so having this information available in the hub would also help address this use case.

[davanstrien]

Adding a relevant paper related to this topic: TimeLMs: Diachronic Language Models from Twitter

[severo]

Related: https://github.com/huggingface/datasets/issues/3877

[severo]

Also related: the Data Catalog Vocabulary - DCAT standard will be discussed in a new Working Group at the W3C: https://www.w3.org/2022/06/dx-wg-charter.html