flux-framework / dyad

DYAD: DYnamic and Asynchronous Data Streamliner
GNU Lesser General Public License v3.0
7 stars 5 forks source link

Adding a use case example 'shuffle' #15

Open JaeseungYeom opened 1 year ago

JaeseungYeom commented 1 year ago

Use case demonstration for input file shuffling with deep learning training. DYAD needs to be updated to leverage fcntl() to dynamically determine the mode of each individual open file, and handle it accordingly. This code is independent of DYAD and Flux. However, it reuses some of utility functions in DYAD implementation. The source files of those are symbolically linked to avoid duplication.

ilumsden commented 3 months ago

What's left to do: