search

qsbase / qs

Quick serialization of R objects
405 stars 19 forks source link

Reading specific chunks from disk #70

Closed gevro closed 2 years ago

gevro commented 2 years ago

Hi, I have a very large data object, made by analyzing a large dataset. The large data object is in the same format every time. However, it is too big to load all into memory.

Is there a way to read chunks of an RDS object from disk, without having to load the whole object into memory? Then I could process each chunk separately.

It would also be helpful to be able to split into chunks and merge chunks.

Thanks.

traversc commented 2 years ago

Sorry, it's not possible. I would suggest splitting up your results into multiple chunks/files.