search

DiskFrame / disk.frame

Fast Disk-Based Parallelized Data Manipulation Framework for Larger-than-RAM Data
https://diskframe.com
Other
594 stars 40 forks source link

Cannot convert/write disk.frame to disk if tibble contains list columns #169

Open b-rodrigues opened 5 years ago

b-rodrigues commented 5 years ago

Reproducible code:

library(tidyverse)
library(disk.frame)

data(starwars)

as.disk.frame(starwars) #fails because fst does not support list-columns yet

as.disk.frame(select_if(starwars, ~!is.list(.))) #works

This is not a bug of disk.frame per se, as this is a shortcoming of fst, but it might be worthwile to show a message to the user stating this. It is entirely possible that other column types are not supported as well.

xiaodaigh commented 5 years ago

Thanks for this. At some point disk.frame will support multiple back ends. So an rds backend would help solve this. But it will happen after documentation and testing and a few big features. @b-rodrigues Thanks for your helpful suggestion.

GitHunter0 commented 3 years ago

@xiaodaigh , I'm really looking forward to .rds, .parquet and .feather (which is now the Apache Arrow standard) backends, it would be great. Thank you for the awesome package