flux-framework / dyad

DYAD: DYnamic and Asynchronous Data Streamliner
GNU Lesser General Public License v3.0
7 stars 5 forks source link

Feature/file lock #61

Closed JaeseungYeom closed 7 months ago

JaeseungYeom commented 7 months ago

Protect the writing of locally cached file by consumer processes. This PR is based on #59 When two consumers attempt to fetch the same file at the same time, each of them will first attempt to open the local copy of the file in write mode. Then, it places an exclusive lock for writing. If there already exists a lock in place, the locking call waits until the existing lock is released. Upon entering the critical region, the consumer process will check if the file size is greater 0, which means that the file has already been written by another process. Then, it moves on without fetching the remote file. If the file size is 0, then, it fetches the remote file into memory and write it into the file open. Finally, it releases the lock.

The process is as follows:

  1. data_get_data() is taken out of dyad_pull(). Then, data_pull() has been renamed to data_cons_store().
  2. added utility functions for locking and unlocking a file
  3. added a utility function to check the size of a file via the file descriptor
  4. added extra states to the dyad context: node index and process id