search

MIT-LCP / physionet-build

The new PhysioNet platform.
https://physionet.org/
BSD 3-Clause "New" or "Revised" License
56 stars 20 forks source link

Expand Schema.org metadata to support Croissant 🥐 #2184

Open tompollard opened 9 months ago

tompollard commented 9 months ago

In https://github.com/MIT-LCP/physionet-build/pull/640 we added support for Schema.org metadata. Croissant is an extension to this schema that captures additional information for finding and using the underlying data (with a focus on machine learning research).

The Croissant project is hosted at: https://github.com/mlcommons/croissant. There are example datasets at: https://github.com/mlcommons/croissant/tree/main/datasets (see "metadata.json" for the relevant dataset).

There are notes on working with Croissant in TensorFlow at: https://www.tensorflow.org/datasets/format_specific_dataset_builders#croissantbuilder

We should consider expanding our Schema.org metadata to conform with the Croissant spec.

Geoff Thomas (Kaggle) suggested that we may encounter issues around file size (e.g. apparently metadata files can get very large, to the point where Google Datasets refuses to index them).

tompollard commented 9 months ago

There are MVP guidelines for Croissant implementation at: https://docs.google.com/document/d/1GvlOUI4FmqsVyysJp6RrONnvSkkOOC1MRpomai3XvBU/ (protected document, request access if needed)