LLNL / UnifyFS

UnifyFS: A file system for burst buffers
Other
102 stars 31 forks source link

Migrating data between Unifyfs servers #728

Open hariharan-devarajan opened 2 years ago

hariharan-devarajan commented 2 years ago

Describe the problem you're observing

In the case of producer-consumer workloads where a subset of producers produce and then another subset of processes on a different set of nodes consume, currently the reading of consumers is bottlenecked by the unifyfs server which holds the data (generally the set of nodes where data was produced).

We can add a migration step between the unifyfs servers to migrate this data closer to the consumers so that they can access data node-locally instead of remote access.