search

posit-dev / positron

Positron, a next-generation data science IDE
Other
2.38k stars 70 forks source link

Data Explorer: implement DuckDB for backend data viewer data #1602

Closed jgutman closed 6 months ago

jgutman commented 11 months ago

Currently, the backend data for use in the data viewer is stored in memory by the Python and R language engines, and any operations we might want to perform on the data (sorting, filtering, not yet implemented) would have to be implemented separately in each language engine. We'd like to set up DuckDB for the backends (do we want in-memory only or write temp files to disk) and use its Rust/Python APIs and use SQL query like syntax to perform the database-like operations the data viewer will eventually need to implement (sorting and filtering first and foremost, perhaps grouping/aggregation down the line?). As we implement live updates for the data viewer, we'd also need to allow for taking these updates from the language backend in the duck db instance

This is likely a pre-requisite for #1077 which is considered a must-have for either internal preview or beta (not sure)

jthomasmock commented 6 months ago

This was originally targeted for the Data Viewer, but we have moved forward with a language-based runtime for in-memory data and #2187 covers the language-agnostic file-on-disk viewer.

@wesm - do you think we can close this in favor of #2187?

wesm commented 6 months ago

Yes, sounds good. Thanks all!