flux-framework / dyad

DYAD: DYnamic and Asynchronous Data Streamliner
GNU Lesser General Public License v3.0
7 stars 5 forks source link

File cache replacement and prefetching #85

Open JaeseungYeom opened 8 months ago

JaeseungYeom commented 8 months ago

We can add LRU replacement scheme as a reference and for general use cases. However, for deep learning use cases, LRU many not be as appropriate as other cases. With DL, we first design prefetching schemes. Then, based on the prefetching targets, we select eviction targets. If a file that we want to prefetch is already present, we should not evict it and avoid prefetching. This issue will need to be addressed by multiple sub plans.