Use smallest feasible data type for CSV columns to avoid MemoryError

EscVM / OIDv4_ToolKit

Download and visualize single or multiple classes from the huge Open Images v4 dataset

GNU General Public License v3.0

800 stars 633 forks source link

Use smallest feasible data type for CSV columns to avoid MemoryError #28

Closed mikkleini closed 5 years ago

mikkleini commented 5 years ago

I was trying to get Openimages dataset in Google cloud server. I increased RAM up to 13GB, reduced threads down to 2 and still got the MemoryError on 64-bit Python. And then i found a suggestion from here to reduce CSV data types: https://www.kaggle.com/c/facebook-v-predicting-check-ins/discussion/20918

So this pull request helps solving https://github.com/EscVM/OIDv4_ToolKit/issues/15

keldrom commented 5 years ago

@mikkleini thank you for this suggestion. It will be very useful.