Allow bulk visit creation

mozilla / chronicle

find everything you've ever found

http://mozillachronicle.tumblr.com/

Mozilla Public License 2.0

16 stars 6 forks source link

Allow bulk visit creation #217

Open nchapman opened 9 years ago

nchapman commented 9 years ago

When uploading past history from the add-on, it'd be much more efficient to submit these in bulk. We should add a new visits endpoint or modify the POST /v1/visits endpoint to support this.

pdehaan commented 9 years ago

Was the current plan to just upload the past 500+/- history items? Or just for the past X days?

nchapman commented 9 years ago

I'm thinking that we'll work towards getting everything at some point. Right now that's a good way to burn through embedly API requests, so we'll probably limit it a bit for the next month until we're confident with our content extraction strategy.

pdehaan commented 9 years ago

Interesting. After a bit of poking and searching, I found out where Google Chrome stores my browsing history (~/Library/Application Support/Google/Chrome/Default — but I had to run $ chflags nohidden ~/Library for that to show in my Finder).

Then I opened the 19MB file in SQLite Professional and it looks like I have 33,606 rows in my "visits" table, and 14,058 rows in my "urls" table. I haven't tried exporting that mess into JSON yet or something more friendly than SQLite.

It looks like my Bookmarks file is a 48KB nested JSON format blob, so I'll have to do a bit of magic to try and extract the number of bookmarks I have. I don't imagine it'd be many since I don't really use bookmarks apart from random intranet pages.

pdehaan commented 9 years ago

In Firefox (at least FF Beta), relevant paths are:

~/Library/Application\ Support/Firefox/Profiles/ << PROFILE >> /places.sqlite (about 10MB in my case)

Not sure how to read any of these tables, but marginally relevant stats seem to be:

moz_bookmarks table: 131 rows
moz_favicons table: 592 rows
moz_historyvisits table: 17,192 rows
moz_places rable: 7,591 rows

jaredhirsch commented 9 years ago

I bet we can get by for a long while with 20-40 imported visits, enough to get a couple pages' worth of UI onto the screen.

jaredhirsch commented 9 years ago

firefox might have something relevant: https://developer.mozilla.org/en-US/Add-ons/SDK/Low-Level_APIs/places_history

chrome seems to have the nicer API here, I wonder if anything has been built to match its niceness, but maybe not yet documented on MDN: https://developer.chrome.com/extensions/history