search

marimo-team / marimo

A reactive notebook for Python — run reproducible experiments, execute as a script, deploy as an app, and version with git.
https://marimo.io
Apache License 2.0
5.39k stars 158 forks source link

Notebooks with large dataframes hang when opening #1586

Closed akshayka closed 3 weeks ago

akshayka commented 3 weeks ago

Hi there,cool for implementing that new feature!

Could it be, that the datasources panel automaticly retrieves informations from every datasource? As an Ibis-table can be a connection to a very large databasetable this could take a while and it would be nice to surpress this on demand.

With the change from #1522, my old notebook hangs when opening

Originally posted by @szst11 in https://github.com/marimo-team/marimo/issues/1522#issuecomment-2158671559

mscolnick commented 3 weeks ago

@szst11, it gets information for only tables that are declared as variables. it also only grabs the following information:

however, for ibis, we do convert it to a pyarrow table which could be the expensive portion. we can look into just loading the schema, and lazily get the rows/columns when asked