jTaylo99 / kolumnextractor

A data extractor designed around building your requirements and easily extracting and verifying your data.
4 stars 8 forks source link

Make the reader an iterator #39

Open geoffleyland opened 1 year ago

geoffleyland commented 1 year ago

Currently, reading a CSV involves reading a whole CSV file, and then transforming it in memory. This is not ideal. A better design would be to return a row iterator (which is what you get from the vanilla CSV reader) and let the user iterate over the lines of a file. If they want to pipe it into an array or dataframe, an iterator makes that easy. If they don't, they don't have to suffer the overhead of multiple copies of the file in memory at once.