search

scraperwiki / spreadsheet-download-tool

A ScraperWiki plugin for downloading data from a box as a CSV or Excel spreadsheet
BSD 2-Clause "Simplified" License
2 stars 1 forks source link

Make streaming friendly #45

Closed pwaller closed 10 years ago

pwaller commented 10 years ago

The memory usage of the input was massive since it was holding the whole input, the whole XML parse tree and the whole table which was about to be written.

Now we only have the current row in memory.

pwaller commented 10 years ago

I think at this point the unicodecsv writer is streaming correct.

The next thing will be to make the excel output streaming friendly, since it's currently not. For that we'll probably want to use PyExcelerate or maybe even write out the xlsx ourselves.