Browsing Activity / History

infogulch commented 2 years ago

1. What is the data source you want to add?

One of the most interesting sources of my activity, and one that I certainly wish I had more insight into is right in my browser: which sites I visit, from where, and how much time I spend on each.

2. What constitutes an "item" from this data source?

A "Page Visit", consisting of: open timestamp, leave timestamp, active duration, url, browser session (device, window), optional referrer (if opened via link) either url or another Page Visit.

3. How are items obtained from the data source?

A few options:

A browser extension that records the data. This would be similar to promnesia which does exactly this.
An export of browser history taken manually. This would be lower fidelity because it doesn't track referrer data or timestamp/duration information in much detail. Less intrusive and less development work than 1, but requires ongoing manual effort from the user to export and import.
- Firefox:
  - Export directly: http://www.nirsoft.net/utils/mozilla_history_view.html
  - Query directly from places.sqlite in .default profile. See How do I export the Browsing History in Firefox?
- Chrome
  - Export directly: http://www.nirsoft.net/utils/chrome_history_view.html
  - Query from History.sqlite in default profile. See: https://www.reddit.com/r/chrome/comments/lqh0mg/is_there_a_way_to_export_chrome_history/
  - Export Chrome History extension: https://chrome.google.com/webstore/detail/export-chrome-history/dihloblpkeiddiaojbagoecedbfpifdj?hl=en
Collecting browser history from the browser's sync service directly. Same limitations as 2, but more automatic.
- My Activity via Google takeout https://myactivity.google.com/myactivity?pli=1
  - Includes youtube and app activity which might also be good to export. See AppUsageStatistics
- Chrome: Sync - https://www.chromium.org/developers/design-documents/sync/
  
  A library that implements the client side of our sync protocol
- Firefox: https://mozilla-services.readthedocs.io/en/latest/sync/index.html

As far as implementation strategy goes, the order should probably be 2, 3, 1. Maybe skip 2 entirely.

3a. If authentication is required, how does a user create or obtain credentials for Timeliner to access the data?

TODO

3b. If an API is available, what are its rate limits?

TODO

3c. If a file is imported, how is the file obtained?

TODO

3d. If a file is imported, how do we read the file?

TODO

4. How can items from this data source be related?

Could link to other Page Visits, or any other item that has a visitable url.

5. What constitutes a "collection" from this data source?

Potential collections: Browser session, Site Visit (collection of related page visits on the same site)

6. What might not be trivial, obvious, or straightforward when implementing this data source?

Maintenance and publishing a whole browser extension to get detailed data.

Given that there are so many options, this issue may need to be split up into separate issues. But I thought I'd open this first to get some input.

mholt commented 2 years ago

Neat idea!

I think we could support multiple ways of getting the browser history, as you mentioned. (Thank you for the linked resources.) So we could have one data source that supports the export format of a browser extension. We could have another data source that supports Firefox exports, another for Chrome exports, etc.

All of them collectively could probably be discussed in this issue.

infogulch commented 2 years ago

I found this gist that explores the firefox places.sqlite history database, including some interesting related queries which seem to indicate that ff keeps more details about navigated-from pages than I originally thought: https://gist.github.com/olejorgenb/9418bef65c65cd1f489557cfc08dde96

See also the firefox source code that also includes some queries: https://searchfox.org/mozilla-esr60/source/toolkit/components/places/nsNavBookmarks.cpp

mholt / timeliner