Updated dataset analyze! to no longer read the entire CSV into memory. - Githubissues

codeforamerica / classifyr

A tool for aggregating and crowd-sourcing the classification emergency call data

MIT License

0 stars 1 forks source link

Updated dataset analyze! to no longer read the entire CSV into memory. #131

Closed jamesiarmes closed 1 year ago

jamesiarmes commented 1 year ago

Overview

Updated dataset analyze! to no longer read the entire CSV into memory.

Type of change

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Related tickets

Fixes # (issue)

Changes

Updated dataset analyze! to no longer read the entire CSV into memory.

Notes

Addresses memory crashes from large CSVs. Also reduces the total analysis time for all files, increasing exponentially with file size.