search

writer / writer-framework

No-code in the front, Python in the back. An open-source framework for creating data apps.
https://dev.writer.com/framework/introduction
Apache License 2.0
1.3k stars 73 forks source link

Maximum state data size #42

Open lianaling opened 1 year ago

lianaling commented 1 year ago

Dear developer,

Thank you so much for this initiative as I have been frustrated by the slowness of Streamlit, especially after setting everything up. I understand that speed is one of the largest advantages of Streamsync over Streamlit, but what about support for large data files? I usually work with large CSV files that measure by GBs (1-5GBs) that I will need to save into the state variable. Is there a known maximum state size for now?

Add-on: I've experienced noticeable lag for loading the Builder after loading ~2GB worth of DataFrame into the state variable.

ramedina86 commented 1 year ago

Hi, thanks for trying Streamsync and sharing your use case. It's very helpful.

There's no set limit for state size. State initialisation will work for larger files, but right now you're restricted by the incremental updates to state. All updates are carried out via WebSockets and the limit is set at 200mb. I'll work on handling larger updates by splitting WebSockets messages.

The next Streamsync release which uses Apache Arrow for dataframe serialisation will improve the handling of larger files, so the initial lag will be greatly reduced.

lianaling commented 1 year ago

Thank you for your explanation!