Open lianaling opened 1 year ago
Hi, thanks for trying Streamsync and sharing your use case. It's very helpful.
There's no set limit for state size. State initialisation will work for larger files, but right now you're restricted by the incremental updates to state. All updates are carried out via WebSockets and the limit is set at 200mb. I'll work on handling larger updates by splitting WebSockets messages.
The next Streamsync release which uses Apache Arrow for dataframe serialisation will improve the handling of larger files, so the initial lag will be greatly reduced.
Thank you for your explanation!
Dear developer,
Thank you so much for this initiative as I have been frustrated by the slowness of Streamlit, especially after setting everything up. I understand that speed is one of the largest advantages of Streamsync over Streamlit, but what about support for large data files? I usually work with large CSV files that measure by GBs (1-5GBs) that I will need to save into the
state
variable. Is there a known maximum state size for now?Add-on: I've experienced noticeable lag for loading the Builder after loading ~2GB worth of DataFrame into the state variable.