Closed kuwisdelu closed 1 month ago
Due to the processing overhead of subsetting a matter
object (i.e., decompressing and re-compressing the atoms
) this is probably not worth it relative to the overhead of serializing a single chunk.
Closing for now.
When
TRUE
, we serialize the data frommatter
objects to workers.When
FALSE
, we only serialize the metadata and the data is loaded locally on the workers. (Note: This requires the host and workers see the same paths to the data files!)