search

dogsheep / google-takeout-to-sqlite

Save data from Google Takeout to a SQLite database
Apache License 2.0
105 stars 8 forks source link

killed by oomkiller on large location-history #2

Open khimaros opened 3 years ago

khimaros commented 3 years ago

memory seems to grow unbounded and is oom-killed after about 20GB memory usage.

this is happening while loading a ~1GB uncompressed location history.

simonw commented 3 years ago

The memory profiler tricks I used in https://github.com/dogsheep/healthkit-to-sqlite/issues/7 could help figure out what's going on here.

khimaros commented 3 years ago

it looks like almost all of the memory consumption is coming from json.load().

another direction here may be to use the new "Semantic Location History" data which is already broken down by year and month.

it also provides much more interesting data, such as estimated address, form of travel, etc.