mlcommons / storage

MLPerf™ Storage Benchmark Suite
https://mlcommons.org/en/groups/research-storage/
Apache License 2.0
61 stars 19 forks source link

Generating a Cosmoflow dataset is very slow #66

Open Linzsd opened 1 month ago

Linzsd commented 1 month ago

When I was generating the dataset for Cosmoflow, the generation was slow, and it took a minute interval to print the logs once. When I switch the data format to npz, the generation is very fast, why is that?