A human readable time format for the history... to use fzf to search history!

[Twix53791]

Hi,

I am just leaving here a small stone in case of oneday, some of the devs of qutebrowser think about redesigning the database history.

I wrote a small plugin to display the browser history with fzf in a pop up window. It is fast and practicable. Very fast. But to reach this speed, it needs to have minimal operations to make. The problem is that the qutebrowser history database store the time in an epoch time format. It is not practical to search for a date: a year, a month, a day... And rebuilding the time in a human readable format from the epoch time in the plugin obviously slows it a lot, especially when the history contains thousands of urls.

So, I made a patch modifying history.py file in qutebrowser, adding to the table 'CompletionHistory' of history.sqlite the column 'human_time', which stote the time in a format "%Y %B %d %H:%M:%S". So you can limit the scope of the history display by year, then month, then day, then the time...

The problem with this modification is than it breaks if the history database already exist, because the column 'human_time' doesn't exist. I tried a bit to find a way letting qutebrowser to add the column if it doesn't exist, but it seems complicated. So, it is not possible to implement the change in qutebrowser without broking the config of everyone or making important changes in sql.py. That's why I made a patch, and a plugin.

EDIT: thanks to @toofar, I learned than sqlite3 has a tool to output an epoch time format in a human readable form.

For those interested, I also made a patch for tabbedbrowser.py recording the last 100 closed tabs in a file in the same format, with a human readable time format, to use it also with fzf.

@@ -8,6 +8,7 @@
 import functools
 import weakref
 import datetime
+import shutil
 import dataclasses
 from typing import (
     Any, Deque, List, Mapping, MutableMapping, MutableSequence, Optional, Tuple)
@@ -19,7 +20,7 @@
 from qutebrowser.keyinput import modeman
 from qutebrowser.mainwindow import tabwidget, mainwindow
 from qutebrowser.browser import signalfilter, browsertab, history
-from qutebrowser.utils import (log, usertypes, utils, qtutils,
+from qutebrowser.utils import (standarddir, log, usertypes, utils, qtutils,
                                urlutils, message, jinja, version)
 from qutebrowser.misc import quitter, objects

@@ -519,6 +520,23 @@
                 else:
                     self.undo_stack[-1].append(entry)

+                # Patch closed-tabs-history for qute-plugin-fzfhistory
+                closed_history=standarddir.data() + "/closed-tabs-history"
+                tmp_ch="/tmp/qute_closed_history_tmp"
+                url_str = tab.url().toDisplayString() + "\n"
+                now=datetime.datetime.now().strftime("%Y %B %d %H:%M:%S")
+
+                with open(closed_history, 'r') as chf:
+                    with open(tmp_ch, "w") as tmpf:
+                        # If the url already saved, delete the old entry
+                        for line in (chf.readlines() [-99:]):
+                            if url_str not in line:
+                                tmpf.write(line)
+                        # Append the file with the current closed tab data
+                        tmpf.write(now + "|" + tab.title() + "|" + url_str)
+
+                shutil.copy(tmp_ch, closed_history)
+
         tab.private_api.shutdown()
         self.widget.removeTab(idx)

[Earnestly]

Why does this need to change the database? You should be using whatever strptime format you desire given last_atime from the unix epoch when generating completion entries for your tooling.

[Twix53791]

Well, if you find a way to be as fast as sqlite3 reading the database, I am hearing... Currently, my script takes 0.2 sec to display the list of a 10 000 lines history in fzf, and it is because I want a beautifull colored output, it could be faster. The history is growing fast, in a couple of years, i will have much more entries, and the time will not increase much, because sqlite3 and fzf are almost as much fast as reading 1 million entries than 1000. CLI programs loose all their benefits when they are slow. Whithout the lighting speed of sqlite3+fzf, rather use the native qutebrowser feature...

But I think everyone has to recognize the native qutebrowser history display is not practicable. You can't search into the all database straightaway, you can't search for an old date ; and I was missing my old Firefox. But with my fzf tool, it is firefox which is old-fashioned! Realtime search in as much as entries I will have in my entire life, search indiscriminately the page title, the date, the url, can open in one key dozens of urls... Just amazing.

Before, I tried two differents ways, maybe there is a third, I am not a pro dev and not good at python...

• maintain a separate database. The problem is: who update it, and when? If it is not qutebrowser which add, everytime it opens an url, the database (in history.py), which program and when gonna update the separate database? The plugin when it is called. Proble: it has to update all the entries before the last call, sometimes hundreds. And it slows the startup time... a lot!
• I tried to see how it was possible to run an external script every time qutebrowser open an url, a script writing the history in my format in a separate database. But it seems this is not possible for now in qutebrowser. Maybe one day it will have signals and hooks available in external scripts, but for now I can't see a way...

[The-Compiler]

The history database format is an implementation detail. If external tools can work with it, that's nice, but this isn't guaranteed. As such, I don't think it's reasonable to add data to it that only caters to external tools.

[toofar]

@Twix53791 you can use sqlite to convert the unix time in the DB to a string format. Eg select datetime(last_atime, 'unixepoch') from CompletionHistory. In my case doing time sqlite3 history.sqlite "select datetime(last_atime, 'unixepoch') from CompletionHistory" | wc -l for a 300k table I'm seeing 100ms for datetime(...) vs 60ms for just the plain column.

[Twix53791]

Thanks a lot @toofar ! I did'nt know this possibility, its amazing! No need so to change the database history format. I updated my plugin.