Closed timothy-barry closed 2 years ago
readr
seems to be suffering from serious memory leak issues; use instead fread
from data.table
.
Plan: first use split
to split an mtx
file into smaller chunks. Then, read each chunk via fread
. Delete each file after it is read into memory.
The
create_ondisc_matrix_from_mtx
function appears to use a lot of memory when thechunked
option is set to true; this likely is due to a bug inread_delim_chunked
. Consider different approaches to implementing this functionality (e.g., Linuxsplit
) if the bug is not fixed.