search

ubvu / bytes-and-bites

https://ubvu.github.io/bytes-and-bites/
6 stars 1 forks source link

memory efficient data science #17

Open peterstol opened 9 months ago

peterstol commented 9 months ago

common problem: I need a bigger memory as my data does not fit and I get memory errors.

(bad) solution: buy a bigger machine solutions (python) oa: chunking, lazy API (polars vs pandas), parquet instead of cvs,...

ref: https://pythonspeed.com/articles/polars-memory-pandas/