Searchable BI Server reaches MemoryError

[boonhapus]

First Stop

• [X] I have read and agreed to the ThoughtSpot Dev Tools EULA.
• [X] I understand that if I have a critical issue with ThoughtSpot, I should reach out to my account team instead of cs_tools.
• [X] I searched the cs_tools documentation, with the integrated search.

Description

cs_tools tools searchable bi-server uses the Search middleware which iterates over the Search Data API. Search Data API only returns 100K rows of data with each pull. We hold all this data in memory until it needs to be dumped to the syncer. That's no bueno.

We need to explore writing to ts.config.temp_dir instead.

https://github.com/thoughtspot/cs_tools/blob/cd29463c9eb3a92986b194617dd574a9554479a9/cs_tools/api/middlewares/search.py#L148-L159

[boonhapus]

This is resolved as part of V1.4.0 . Check the release notes to learn more.

https://thoughtspot.github.io/cs_tools/changelog/1.4.0