search

seanbreckenridge / HPI

Human Programming Interface - a way to unify, access and interact with all of my personal data [my modules]
https://beepb00p.xyz/hpi.html
MIT License
72 stars 6 forks source link
data gdpr history lifelogging personal-api quantified-self

readme

TLDR: I'm using HPI(Human Programming Interface) package as a means of unifying, accessing and interacting with all of my personal data.

It's a Python library (named my), a collection of modules for:

Why?

This is built on top of karlicoss/HPI. It started out as a fork, but has since been converted to my own set of modules. This is installed alongside the upstream repository (meaning you can use both modules from upstream and here simultaneously), see #install

My Modules

'Historical' Modules

These are modules to parse GDPR exports/data from services I used to use, but don't anymore. They're here to provide more context into the past.

See here for my HPI config

Promnesia Sources for these HPI modules

I also have some more personal scripts/modules in a separate repo; HPI-personal

In-use from karlicoss/HPI

Partially in-use/with overrides:

'Overriding' an all.py file means replacing the all.py from upstream repo (this means it can use my sources here to grab more locations/ips, since those don't exist in the upstream). For more info see reorder_editable, and the module design docs for HPI, but you might be able to get the gist by comparing:

Since I've mangled my PYTHONPATH (see reorder_editable), it imports from my repo instead of karlicoss/HPI. all.py files tend to pretty small -- so overriding/changing a line to add a source is the whole point.

Companion Tools/Libraries

Disregarding tools which actively collect data (like ttt/window_watcher) or repositories which have their own exporter/parsers which are used here, there are a couple other tools/libraries I've created for this project:

I also use this in my_feed, which creates a feed of media/data using HPI, live at https://sean.fish/feed/

Ad-hoc and interactive

Some basic examples.

When was I most using reddit?

>>> import collections, my.reddit.all, pprint
>>> pprint.pprint(collections.Counter([c.created.year for c in my.reddit.all.comments()]))
Counter({2016: 3288,
         2017: 801,
         2015: 523,
         2018: 209,
         2019: 65,
         2014: 4,
         2020: 3})

Most common shell commands?

>>> import collections, pprint, my.zsh
# lots of these are git-related aliases
>>> pprint.pprint(collections.Counter([c.command for c in my.zsh.history()]).most_common(10))
[('ls', 51059),
 ('gst', 11361),
 ('ranger', 6530),
 ('yst', 4630),
 ('gds', 3919),
 ('ec', 3808),
 ('clear', 3651),
 ('cd', 2111),
 ('yds', 1647),
 ('ga -A', 1333)]

What websites do I visit most?

>>> import collections, pprint, my.browser.export, urllib
>>> pprint.pprint(collections.Counter([urllib.parse.urlparse(h.url).netloc for h in my.browser.export.history()]).most_common(5))
[('github.com', 20953),
 ('duckduckgo.com', 10146),
 ('www.youtube.com', 10126),
 ('discord.com', 8425),
 ('stackoverflow.com', 2906)]

Song I've listened to most?

>>> import collections, my.mpv.history_daemon
>>> collections.Counter([m.path for m in my.mpv.history_daemon.history()]).most_common(1)[0][0]
'/home/sean/Music/JPEFMAFIA/JPEGMAFIA - LP! - 2021 - V0/JPEGMAFIA - LP! - 05 HAZARD DUTY PAY!.mp3'

Movie I've watched most?

>>> import my.trakt, from collections import Counter
>>> Counter(e.media_data.title for e in my.trakt.history()).most_common(1)
[('Up', 92)]  # (the pixar movie)

hpi also has a JSON query interface, so I can do quick computations using shell tools like:

# how many calories have I eaten today (from https://github.com/seanbreckenridge/ttally)
$ hpi query ttally.__main__.food --recent 1d -s | jq -r '(.quantity)*(.calories)' | datamash sum 1
2258.5

Install

For the basic setup, I recommend you clone and install both directories as editable installs:

# clone and install upstream as an editable package
git clone https://github.com/karlicoss/HPI ./HPI-karlicoss
python3 -m pip install --user -e ./HPI-karlicoss

# clone and install my repository as an editable package
git clone https://github.com/seanbreckenridge/HPI ./HPI-seanb
python3 -m pip install --user -e ./HPI-seanb

Editable install means any changes to python files reflect immediately, which is very convenient for debugging and developing new modules. To update, you can just git pull in those directories.

If you care about overriding modules, to make sure your easy-install.pth is ordered correctly:

python3 -m pip install --user reorder_editable
python3 -m reorder_editable reorder ./HPI-seanb ./HPI-karlicoss

Then, you likely need to run hpi module install for any modules you plan on using -- this can be done incrementally as you setup new modules. E.g.:

(The install script does that for all my modules, but you likely don't want to do that)

Its possible to install both my packages because HPI is a namespace package. For more information on that, and some of the complications one can run into, see reorder_editable, and the module design docs for HPI.

If you're having issues installing/re-installing, check the TROUBLESHOOTING_INSTALLS.md file.

If you recently updated and it seems like something has broke, check the CHANGELOG for any possible breaking changes