Closed gruuya closed 1 year ago
This is so as to avoid accumulating the entire result in memory and potentially OOM-ing the process. Closes #258
Example, using a large table to perform SELECT * ... LIMIT 100000:
SELECT * ... LIMIT 100000
W/O streaming memory peaks at 220MB (current main)
W/ streaming memory peaks at 30MB (this PR)
This is so as to avoid accumulating the entire result in memory and potentially OOM-ing the process. Closes #258
Example, using a large table to perform
SELECT * ... LIMIT 100000
:W/O streaming memory peaks at 220MB (current main)![slika](https://github.com/splitgraph/seafowl/assets/45558892/39c398b2-f582-49d7-a81d-1f090c727d38)
W/ streaming memory peaks at 30MB (this PR)![slika](https://github.com/splitgraph/seafowl/assets/45558892/804f8796-42a5-422c-8754-3d915e1fcfea)