search

flux-framework / dyad

DYAD: DYnamic and Asynchronous Data Streamliner
GNU Lesser General Public License v3.0
7 stars 5 forks source link

Use Flux KVS "ver" field to help determine when to transfer files #46

Open ilumsden opened 11 months ago

ilumsden commented 11 months ago

The Flux KVS keeps track of the "version" of its entries through a "ver" field. We should use this to help determine if a file has changed. If the version of the KVS entry changes, we transfer. Otherwise, we don't.

ilumsden commented 11 months ago

Alternatively, we could do the following:

  1. Append/prepend the version number to the file name
  2. Transfer data and save under new file name
  3. Cache file name and version number for future lookup
  4. Return handle to file w/ new name
chu11 commented 11 months ago

The Flux KVS keeps track of the "version" of its entries through a "ver" field. We should use this to help determine if a file has changed. If the version of the KVS entry changes, we transfer. Otherwise, we don't.

Just an FYI, ver in KVS entries probably isn't what you think it is. It's the version of the tree object format that data is stored in. You can see it doesn't get updated in this example ...

>flux kvs put foo=1
>flux kvs get --treeobj foo
{"data":"MQ==","type":"val","ver":1}
>flux kvs put foo=blarg
>flux kvs get --treeobj foo
{"data":"Ymxhcmc=","type":"val","ver":1}