kuwisdelu / matter

Out-of-core statistical computing and signal processing
http://www.cardinalmsi.org
54 stars 4 forks source link

Add `serialize = TRUE` option to chunk apply functions #29

Closed kuwisdelu closed 1 month ago

kuwisdelu commented 2 months ago

When TRUE, we serialize the data from matter objects to workers.

When FALSE, we only serialize the metadata and the data is loaded locally on the workers. (Note: This requires the host and workers see the same paths to the data files!)

kuwisdelu commented 1 month ago

Due to the processing overhead of subsetting a matter object (i.e., decompressing and re-compressing the atoms) this is probably not worth it relative to the overhead of serializing a single chunk.

Closing for now.