search

act-now-coalition / can-scrapers

MIT License
9 stars 13 forks source link

Use `dtypes` when loading db csv to reduce memory usage #448

Closed smcclure17 closed 1 year ago

smcclure17 commented 1 year ago

For a 1,000,000 row slice of the database view this reduced the loaded dataframe size from 128MB to 55MB. Hopefully this will also decrease the amount of memory needed for the parquet file updater.

E.g.:

image