Quick links: [[#motivation][Motivation]] | [[file:doc/GUIDE.org#extension-features][Feature guide]] | [[#demos][Demos]] | [[#how-does-it-work][How does it work?]] | [[#install][Install]] | [[#try-it-out][Try it out & Setup]] | [[#glossary][Glossary]] | [[file:doc/GUIDE.org#FAQ][FAQ]] | [[#support][Support]] | [[file:doc/TROUBLESHOOTING.org][Troubleshooting guide]] | [[file:doc/DEVELOPMENT.org][Developer's guide]]
Promnesia is a browser extension for [[https://chrome.google.com/webstore/detail/promnesia/kdmegllpofldcpaclldkopnnjjljoiio][Chrome]]/[[https://addons.mozilla.org/en-US/firefox/addon/promnesia][Firefox]] (including Firefox for Android!) which serves as a web surfing copilot, enhancing your browsing history and web exploration experience.
TLDR: it lets you explore your browsing history in context: where you encountered it, in chat, on Twitter, on Reddit, or just in one of the text files on your computer. This is unlike most modern browsers, where you can only see when you visited the link.
It allows you to answer different questions about the current web page:
See [[file:doc/GUIDE.org#extension-features][feature guide]] for more information.
You can jump straight to the [[#demos][Demos]] and [[#install][Install]] sections if you're already overexcited.
Motivation See a [[https://beepb00p.xyz/promnesia.html][writeup]] on history and the problems Promnesia is solving. Here's a short summary.
Has it ever occurred to you that you were reading an old bookmark or some lengthy blog post and suddenly realized you had read it already before? It would be fairly easy to search in the browser history, however, it is only stored locally for three months.
Or perhaps you even have a habit of annotating and making notes elsewhere? And you wanna know quickly if you have the current page annotated and display the annotations. However, most services want you to use their web apps even for viewing highlights. You don't know if you have highlights, unless you waste time checking every page.
Or you have this feeling that someone sent you this link ages ago, but you don't remember who and where.
Or you finally got to watch that thing in your 'Watch later' youtube playlist, that's been there for three years, and now you want to know why did you add it in the first place.
Then this tool is for you.
See [[file:doc/GUIDE.org#extension-features][feature guide]] if you're wondering how something specific here works.
[[https://karlicoss.github.io/promnesia-demos/screens/promnesia andy Screenshot at 2020-06-05 23-33-07.png]]
Normally in order to see see your Instapaper highligths you have to go to the web app first!
If I click =notes/ip2org.org=, it will cause my Emacs to open and jump to the file, containing this note.
[[https://karlicoss.github.io/promnesia-demos/screens/promnesia universal love Screenshot at 2020-06-05 23-18-38.png]]
Clicking on the link will open Telegram and jump straight to the message where the link was mentioned. So you can reply to it without having to search or scroll over the whole chat history.
Similarly, clicking would jump straight to my tweet.
Annotated demo of sidebar with direct & child visits, and 'visited' marks (click to zoom)
Annotated demo of highlight and locator features (click to zoom)
More:
A [[https://www.youtube.com/watch?v=9PsOeYheIY4][video]], demonstrating
[[file:doc/GUIDE.org#mark-visited]['mark visited']] feature
highlights
jumping to the originating tweet; jumping straight to the org-mode heading where links occurred
You can find more short screencasts, demonstrating various features [[https://github.com/karlicoss/promnesia-demos][here]]
There are more screenshots [[https://github.com/karlicoss/promnesia/issues/5#issuecomment-619365708][here]]
This [[https://beepb00p.xyz/myinfra-roam.html#promnesia][post]] features a video demo of using Promnesia with a Roam Research database
Video demos for the mobile extension:
[[https://karlicoss.github.io/promnesia-demos/mobile/panel-w3c.webm][showing all references to pages on W3C website in my org-mode notes]]
[[https://karlicoss.github.io/promnesia-demos/mobile/mark-visited-reddit.webm][marking already visited links on the page]], to make easier to process Reddit posts
How does it work? :PROPERTIES: :CUSTOM_ID: how-does-it-work :END: Promnesia consists of three parts:
browser extension
However, browser addons can't read access your filesystem, so to load the data we need a helper component:
server/backend: =promnesia serve= command
It's called 'server', but really it's just a regular program with the only purpose to serve the data to the browser. It runs locally and you don't have to expose it to the outside.
indexer: =promnesia index= command
Indexer goes through the sources (specified in the config), processes raw data and extracts URLs along with other useful information.
Another important thing it's doing is normalising URLs to establish equivalence and strip off garbage. I write about the motivation for it in [[https://beepb00p.xyz/promnesia.html#urls_broken]["URLs are broken"]].
You might also want to skim through the [[https://github.com/karlicoss/promnesia#glossary][glossary]] if you want to understand deeper what information Promnesia is extracting.
** Data sources Promnesia ships with some builtin sources. It supports:
data exports from online services: Reddit/Twitter/Hackernews/Telegram/Messenger/Hypothesis/Pocket/Instapaper, etc.
It heavily benefits from [[https://github.com/karlicoss/HPI][HPI]] package to access the data.
Google Takeout/Activity backups
Markdown/org-mode/HTML or any other plaintext on your disk
in general, anything that can be parsed in some way
you can also add [[https://github.com/karlicoss/promnesia/blob/master/doc/SOURCES.org#extending][your own custom sources]], Promnesia is extensible
See [[https://github.com/karlicoss/promnesia/blob/master/doc/SOURCES.org][SOURCES]] for more information.
** Data flow
Here's a diagram, which would hopefully help to understand how data flows through Promnesia.
See HPI [[https://github.com/karlicoss/HPI/blob/master/doc/SETUP.org#data-flow][section on data flow]] for more information on HPI modules and data flow.
Also check out my [[https://beepb00p.xyz/myinfra.html#promnesia][infrastructure map]], which is more detailed!
: ┌─────────────────────────────────┐ ┌────────────────────────────┐ ┌─────────────────┐ : │ 💾 HPI sources │ │ 💾 plaintext files │ │ other sources │ : │ (twitter, reddit, pocket, etc.) │ │ (org-mode, markdown, etc.) │ │ (user-defined) │ : └─────────────────────────────────┘ └────────────────────────────┘ └─────────────────┘ : ⇘⇘ ⇓⇓ ⇙⇙ : ⇘⇘ ⇓⇓ ⇙⇙ : ┌──────────────────────────────┐ : │ 🔄 promnesia indexer │ : | (runs regularly) │ : └──────────────────────────────┘ : ⇓⇓ : ┌──────────────────────────────┐ : │ 💾 visits database │ : │ (promnesia.sqlite) │ : └──────────────────────────────┘ : ⇓⇓ : ┌──────────────────────────────┐ : │ 🔗 promnesia server │ : | (always running) | : └──────────────────────────────┘ : ⇣⇣ : ┌─────────────────────────────────┐ : ┌───────────────────────┤ 🌐 web browser ├────────────────────┐ : │ 💾 browser bookmarks ⇒ (promnesia extension) ⇐ 💾 browser history | : └───────────────────────┴─────────────────────────────────┴────────────────────┘
Install :PROPERTIES: :CUSTOM_ID: install :END:
extension:
It can be useful because Chrome Web Store releases might take days to approve, but in general the store version if preferrable.
backend
Note that Promnesia can work without the backend, so technically this step is optional. But then it will only be able to use browser visits and browser bookmarks, so the benefits of the extension will be limited.
This is mainly useful for tinkering with the code and writing new modules.
You might also need some extra dependencies. See [[file:doc/SOURCES.org#extra-dependencies]["Extra dependencies"]] for more info.
As for supported operating systems:
You might want to run =brew install libmagic= for proper MIME type detection.
Try it out You can try out Promnesia with the extension only, it uses browser history and bookmarks as data sources. However, Promnesia's main power is using data from external sources, and for that you'll need to run the indexer and backend (more on it in the next section).
The easiest way to try this mode is to run =promnesia demo= command, it can give you a sense of what Promnesia is doing with zero configuration.
[[#install][Install]] the extension and the server (PIP package), in case you haven't already
Run ~promnesia demo https://github.com/karlicoss/exobrain~
This clones the repository, ([[https://github.com/karlicoss/exobrain][my personal wiki]] in this case), extracts the URLs, and runs on the port =13131= (default, can be specified via =--port=)
You can also use a path on your local filesystem (e.g. directory with text files), or a website URL.
After that, visit https://www.reddit.com
If you press the extension icon, you will see the pages from my blog where I link to posts on Reddit.
Setup
To get the most benefit from Promnesia, it's best to properly setup your own config, describing the sources you want it to use. If something is unclear, please feel free to open issues or reach me, I'm working on improving the documentation. Also check out [[file:doc/TROUBLESHOOTING.org][troubleshooting guide]] or [[https://github.com/karlicoss/promnesia/labels/documentation][open discussions on documentation]].
create the config: =promnesia config create=
The command will put a stub promnesia config in your user config directory, e.g. =~/.config/promnesia/config.py= on Linux. (it's possibly different on OSX and Windows, see [[https://github.com/ActiveState/appdirs/blob/3fe6a83776843a46f20c2e5587afcffe05e03b39/appdirs.py#L187-L190][this]] if you're not sure). If you wish to specify a custom location, you can set the ~PROMNESIA_CONFIG~ environment variable or pass the ~--config~ flag.
edit the config and add some sources
You can look at an [[file:src/promnesia/misc/config_example.py][example config]], or borrow bits from an annotated configuration example here: [[file:doc/config.py]].
The only required setting is:
SOURCES specifies the list of data sources, that will be processed and indexed by Promnesia.
You can find the list of available sources with more documentation on each of them here: [[file:doc/SOURCES.org][SOURCES]].
If you want to learn about other settings, the best way at the moment (apart from reading [[file:src/promnesia/config.py][the source]]) is, once again, [[file:doc/config.py][example config]].
First, you can run =promnesia doctor config=, it can be used to quickly troubleshoot typos and similar errors. Note that you may need to install mypy for some config checks.
Next, you can use the demo mode: =promnesia demo --config /path/to/config.py=.
This will index the data and launch the server immediately, so you can check that everything works as expected in your browser.
run the indexer: =promnesia index=
[[https://github.com/karlicoss/promnesia/issues/20][At the moment]], indexing is periodic, not realtime. The best is to run it via cron/systemd once or several times a day:
: # run every hour in cron : 0 promnesia index >/tmp/promnesia-index.log 2>/tmp/promnesisa-index.err
Note: you can also pass =--config /path/to/config.py= explicitly if you prefer or want to experiment.
Run =promnesia doctor database= to quickly inspect the database and check that stuff that you wanted got indexed. You might need to install =sqlitebrowser= first.
run the server: =promnesia serve=
You only have to start it once, it will automatically detect further changes done by =promnesia index=.
This sets it up to autostart and run in the background:
I /think/ you can also use cron with =@reboot= attribute:
: # sleep is just in case cron starts up too early. Prefer systemd script if possible! : @reboot sleep 60 && promnesia serve >/tmp/promnesia-serve.log 2>/tmp/promnesia-serve.err
Alternatively, you can just create a manual autostart entry in your desktop environment.
[optional] setup MIME handler to jump to files straight from the extension
See a short [[https://karlicoss.github.io/promnesia-demos/jump_to_editor.webm][20s demo]], and if this is something you'd like, follow the instructions in [[https://github.com/karlicoss/open-in-editor#readme][open-in-editor]].
In code ([[file:src/promnesia/common.py][python]], [[file:extension/src/common.js][JS]]), visits are reprented as =class Visit= (and =class DbVisit=).
Visits have the following fields:
url: hopefully, no explanation needed!
The only required field.
timestamp: when the page was visited
Required, but in the future might be optional (sometimes you don't have a meaningful timestamp).
locator: what's the origin of the visit?
Usually it's a permalink back to the original source of the visit.
For example:
Required, but in the future might be optional. (TODO also rename to 'origin'??)
context: what was the context, in which the visit occurred?
For example:
I usually call a visit without a context 'boring' -- it doesn't contain much information except for the mere fact of visiting the page before. However they are still useful to have, since they fill in the gaps and provide means of tracing through your history.
Optional.
duration: how much we have spent on the page
Somewhat experimental field, at the moment it's only set for Chrome (and often not very precise).
Optional.
Digression: now that you have an idea what is a Visit, you can understand few more things about Promnesia:
Now let's consider some concrete examples of different kinds of Visits:
[[file:src/promnesia/sources/takeout.py][Google Takeout]] indexer
Results in visits with:
There isn't any context for visits from takeout, because it's basically a fancy database export.
[[file:src/promnesia/sources/instapaper.py][Instapaper]] indexer
Generates a visit for each highlight on the page:
[[file:src/promnesia/sources/markdown.py][Markdown]] indexer
Extracts any links it finds in Markdown files:
However, if you do have your own format, it's possible to write your own indexer to properly take them into the account.
Note: this terminology is not set is stone, so if someone feels there are words that describe these concepts better, I'm open to suggestions!
FAQ See [[file:doc/GUIDE.org#FAQ]]
Support The best support for me would be if you contribute to this or my other projects. Code, ideas of feedback -- everything is appreciated.
I don't need money, but I understand it's often easier to give away than time, so here are some of projects that I donate to:
[[https://archive.org/donate][Internet Archive]]
[[https://github.com/hlissner/doom-emacs#contribute][Doom Emacs]]
More links
[[file:doc/TROUBLESHOOTING.org][Troubleshooting guide]]
[[file:doc/SOURCES.org][Documentation on the sources]]
[[file:doc/DEVELOPMENT.org][Developer's guide]]
[[file:doc/PRIVACY.org][Privacy policy]]