improvement: make ibis and dataframe-like tables lazy evaluate

marimo-team / marimo

A reactive notebook for Python — run reproducible experiments, execute as a script, deploy as an app, and version with git.

https://marimo.io

Apache License 2.0

5.39k stars 158 forks source link

improvement: make ibis and dataframe-like tables lazy evaluate #1589

Closed mscolnick closed 3 weeks ago

mscolnick commented 3 weeks ago

Fixes #1586

make ibis and dataframe-like tables lazy evaluate (specifically for num rows)
add DataFrameProtocolTableManager which can be lazier and more performant when displaying in the datasources panel

vercel[bot] commented 3 weeks ago

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
marimo-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 10, 2024 9:19pm
marimo-storybook	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 10, 2024 9:19pm

github-actions[bot] commented 3 weeks ago

🚀 Development release published. You may be able to view the changes at https://marimo.app?v=0.6.18-dev4

szst11 commented 3 weeks ago

I expect, that the num rows operation could already be an intense operation, if the lazy dataframe e.g. contains some JOINs on a large table.

mscolnick commented 3 weeks ago

@szst11, that makes sense. We now should only be fetching the rows when displaying in a table or chart (we need to resolve the DF anyways to display it). We no longer read the rows on startup / to display it in the sidebar.