optimizations for GoogleEventSet, speeding up merging 20+%

purarue / google_takeout_parser

A library/CLI tool to parse data out of your Google Takeout (History, Activity, Youtube, Locations, etc...)

https://pypi.org/project/google-takeout-parser/

MIT License

82 stars 14 forks source link

optimizations for GoogleEventSet, speeding up merging 20+% #68

Closed karlicoss closed 1 month ago

karlicoss commented 1 month ago

All objects from takeout pass through this merging HPI so worth speeding it up

add add_if_not_present method to avoid computing key twice (which is quite expensive!)

This is intended to be used as a replacement for (e.g. in HPI)
```
if event in emitted: 
  continue
emitted.add(event)
yield event
```
With this method, we could rewrite as:
```
if emitted.add_if_not_present(event):
  yield event
```
This could be introduced to hpi with backwards compatibility.
use type directly as key, types are hashable (very tiny speedup, but it also feels more natural anyway)

purarue commented 1 month ago

Thanks ❤️

Will merge when I get home

purarue commented 1 month ago

@karlicoss bumped the version, since changing how to dedupe events might lead to some weird duplicates/errors if people were doing some custom merge with a cachew database with some other standalone export. (edit: actually, probably not...? since the cachew keys havent changed, its the key thats computed in python code. eh, good to push the perf improvements anyways)

feel free to ping me to bump pypi versions if you ever need em

karlicoss commented 1 month ago

Thanks! I'm running off an editable checkout anyway so don't mind pypi as much. I might push a few more changes in the next few days, sorting out some old branches and todos I never got to contribute :)