search

tidyverse / tidyr

Tidy Messy Data
https://tidyr.tidyverse.org/
Other
1.38k stars 417 forks source link

Think about efficient nesting and unnesting in SQL #247

Closed krlmlr closed 7 years ago

krlmlr commented 8 years ago

For the case where a "data" column contains other data frames, nest() and unnest() for SQL tables could be made more efficient with some support by the DBI backend. For nest(), I'm thinking about a lazy operation that adds nest columns to a tbl; the actual nesting would happen in-place when fetching. The DBI backend could provide a primitive that fetches all rows until the value of one of the nest columns changes.

hadley commented 7 years ago

I think that will be out of scope for tidyr for quite some time: we currently only have one DBI backend that supported nested data structures, and bigrquery can't even fully retrieve them yet.