Open rdmolony opened 2 years ago
We need to find the right tool for the job. The BERs are >1GB uncompressed. pandas reads data into memory & so will crash in 1GB of RAM.
pandas
dask
fugue
ibis
vaex
sqlite
See filter-via-* branches at https://github.com/rdmolony/ibsg for workings on:
filter-via-*
We need to find the right tool for the job. The BERs are >1GB uncompressed.
pandas
reads data into memory & so will crash in 1GB of RAM.Options
dask
fugue
ibis
vaex
sqlite