search

AI-Hypercomputer / tpu-recipes

Apache License 2.0
5 stars 0 forks source link

Data Loading from GCS #8

Open shivajid opened 1 week ago

shivajid commented 1 week ago

It would be good to have data loading support from GCS

A similar conversion of C4 TFRecords to WebDatasets or a more sequential file access format to let users plug in different datasets.

miladm commented 1 week ago

@tengyifei to share his thoughts on the value of this feature. I'm not convinced we need to incorporate many options yet. Curious to gather more input.