search

kaneplusplus / bigmemory

126 stars 24 forks source link

Flush in Rcpp #98

Closed dipterix closed 4 years ago

dipterix commented 4 years ago

Hi, I'm experimenting to extend bigmemory package for my own research project. Right now I'm implementing in Rcpp using BigMatrix headers and file back end to manage the data. Just wonderinh what's the recommended way to flush only partial of the data into disk in Rcpp code given data address?

Thanks.

kaneplusplus commented 4 years ago

Hello. I don't think it's currently supported. A flush ensures that what is on disk is the same as what is in memory. However, processing attaching to an unflushed shared memory resource will all have access to the same data. Are you sure you need a partial flush?

dipterix commented 4 years ago

Thanks. I was just wondering if I want to write 1TB data into hard disk, then what's the best way to do it. (suppose 1TB data are generated in C++ code in a stream way, and I have figured it out how to user big matrix accessors, only 16GB free RAM allowed)