Data Loading from GCS - Githubissues

AI-Hypercomputer / tpu-recipes

Apache License 2.0

2 stars 0 forks source link

Data Loading from GCS #8

Open shivajid opened 4 hours ago

shivajid commented 4 hours ago

It would be good to have data loading support from GCS

WebDataSet Pipeline

A similar conversion of C4 TFRecords to WebDatasets or a more sequential file access format to let users plug in different datasets.

miladm commented 4 hours ago

@tengyifei to share his thoughts on the value of this feature. I'm not convinced we need to incorporate many options yet. Curious to gather more input.