Project: Kaggle (Croissant)

cmbz commented 9 months ago

Overview

"Croissant is a high-level format for machine learning datasets that brings together four rich layers."
This issue tracks activities related to our collaboration with the Kaggle Team related to Croissant.

Make datasets easier to find and work with for Machine Learning, at scale and by diverse stakeholders (e.g. AI engineers, AI ethicists[e], compliance managers, interested public)

Croissant is the most convenient and widely used machine-readable format for ML-ready datasets.

[x] https://github.com/IQSS/dataverse/issues/10341
[x] Announce that Croissant is ready for production use. DONE.
[x] https://github.com/IQSS/dataverse.harvard.edu/issues/294
[x] https://github.com/IQSS/dataverse.harvard.edu/issues/294
[ ] https://github.com/IQSS/dataverse/issues/10542
[ ] https://github.com/IQSS/dataverse-frontend/issues/350
[ ] https://github.com/mlcommons/croissant/issues/530
[x] Reindex datasets in Harvard Dataverse so that the "License" search facet is meaningful. Should be done as part of upgrading to 6.3:
- https://github.com/IQSS/dataverse.harvard.edu/issues/285
[ ] Include a facet for the Search API for datasets that only have files that are truly open (no custom terms, no guestbooks).
[ ] Let Kaggle know how many dataset and bytes to expect when copying CC0 dataset from Harvard Dataverse (see notes from 2024-07-18 meeting and Slack)
[ ] Let Kaggle know the best way to see when datasets have changed
[ ] Commit data from Dataverse to Kaggle via CroissantML via a button, as an explicit action from the user. Is this part of a larger story around pushing data to other systems, such as data lakes?

Depending on the outcome of these issues, we may enhance our Croissant implementation to cover additional use cases.

cmbz commented 1 month ago

Croissant exporter added to both Harvard Dataverse and Demo (see https://github.com/IQSS/dataverse.harvard.edu/issues/294)
The Croissant Task Force was informed that adding Dataverse installations to Croissant Online Health should now be unblocked: https://github.com/mlcommons/croissant/issues/530#issuecomment-2305479611
We met with Goeff from Kaggle during tech hours.

cmbz commented 1 week ago

Now that the Croissant exporter is in place and being indexed by Google, Harvard Dataverse is showing Croissant data in Google Dataset Search, as described in a mailing list post with a screenshot.
Emails exchanged with Croissant Task Force about file formats and DDI-CDI.