flux-framework / dyad

DYAD: DYnamic and Asynchronous Data Streamliner
GNU Lesser General Public License v3.0
6 stars 5 forks source link

Consume after kvs pull #77

Closed JaeseungYeom closed 5 months ago

JaeseungYeom commented 5 months ago
  1. Fixes a bug in dyad_consume() that returns the buffer that has never been assigned.
  2. Adds an variant interface of dyad_consume() that relies on the metadata that has already been pulled from KVS
  3. Still need to add a python interface: note that the new interface dyad_consume_w_mdata(..., mdata) deallocates mdata internally. We can change this behavior depending on how we want to write python interface.
JaeseungYeom commented 5 months ago

We should not duplicate the reading part. We should just split the functionality and call the same function.

Have consume_data do the I/O consume can do fetch of metadata and call consume_data

I understand it will make it more consistent. However, it will also needlessly allow redundant KVS traffic even with C/C++ when python is not in use. The original form is the best, and addition of the two calls dyad_get_metadata() and dyad_consume_w_mdata() is a compromise for python. Do we want to apply that to C and C++? For example, the same file is read locally 10 times, the separate interface will increase the KVS traffic by 10 times.

JaeseungYeom commented 5 months ago

What is your opinion on the meta_data deallocation?

JaeseungYeom commented 5 months ago