libffcv / ffcv

FFCV: Fast Forward Computer Vision (and other ML workloads!)
https://ffcv.io
Apache License 2.0
2.79k stars 180 forks source link

ordering of datapoints in beton files #320

Open meghbhalerao opened 1 year ago

meghbhalerao commented 1 year ago

Hello, when we use the ffcv writer class to write the data into the beton files, is the ordering platform independent, i.e. if I write the beton files using one machine vs using another, will the ordering of elements be the same, so that I can index the same elements using the same set of indices across beton files that are written on different machines? I am running some distributed applications across several machines, and hence I was wondering if the above holds. Some of my machines are CPU only so I was wondering if that would make a difference too. Thanks for your help! Megh