roskakori / cutplace

validate data stored in CSV, PRN, ODS or Excel files
http://cutplace.readthedocs.org/
GNU Lesser General Public License v3.0
18 stars 20 forks source link

Reduce memory foot print of CSV reading #32

Closed roskakori closed 12 years ago

roskakori commented 12 years ago

Currently the CSV reader stores the whole file in memory before parsing it. This is a relict of refactoring the whole parser infrastructure several times during the early days of cutplace.

The proper solution would be to use a consumer/producer like for ODS, where the producer would read the CSV in a separate thread.