Fuzzy file opener

[andreikop]

Hi, @bjones1, @vi, @panhejia, @pasnox, @Yajo

I've just pushed to fuzzyopen branch new amazing file opener. It introduced some questions and I'd like to hear your opinion.

At first, to try the file opener:

• Check out fuzzyopen branch and run Enki
• Choose enki/enki directory in the File Browser
• Press Ctrl+M. You'll see the dialog
• Type qtob. You'll see core/queued_message_tool_bar.by.
• Type navinit. You'll see plugins/navigator/__init__.py.
• Press Enter to open the file

When working with familiar project, this way of opening files is incredible quicker than using Ctrl+O dialogue, Localtor or File browser.

The problems:

Project concept and project root

To fuzzy search files the opener must know project root, from which start to search. Now I use FileBrowser root. But It is temporary solution to try the opener. Something better must be invented.

Sublime and Atom have concept of a project. A project is explicitly opened by the user and fuzzy openers search for files in the project directory. Project tree is also shown by the file browser, and it is impossible to choose another file browser root. I haven't introduced the Project concept in Enki before, because an user have to spend time to explicitly open a project. And because I like very much movable File browser root. But probably now is the time. With fuzzy file opener it is not so necessary to move project root.

There are some other advantages of having project concept. I.e. @vi wants very much to have named sessions and proposed idea of choosing the session with command line switch or environment variable. But both ways not always are comfortable. If we have project, a session can be associated with it. So, when project is opened, Enki reopens opened files. Atom and Sublime do this way.

The keyboard shortcut and the dialogues count

I temporary assigned Ctrl+M shortcut for the dialog, because it is one of the 3 available ctrl-sequences. But, now there are 3 dialogs connected to navigation: Locator, Go To Line and Fuzzy opener. Should it be merged to 2? Or even to 1?

[vi]

I'll check this out (+ cherry-pick 135d88988f4f911c4384616157cac639ce2779a3).

My current annoyance with Enki is that when there are a lot of opened files in the session (and I'm running 4 sessions at the moment), it's inconvenient to find the file in "Opened files" pane, there's no "quick search" for it.

[vi]

Started, pressed Ctrl+M. It started thrashing HDD and displaying "loading".

Fortunately, without UI freeze.

Suddenly:

QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
CRITICAL:root:Traceback (most recent call last):
  File "/mnt/src/git/enki/bin/../enki/plugins/fuzzyopen/fuzzyopen.py", line 77, in run
    self.itemsReady.emit(results)
RuntimeError: wrapped C/C++ object of type ScannerThread has been deleted
CRITICAL:root:Traceback (most recent call last):
  File "/mnt/src/git/enki/bin/../enki/plugins/fuzzyopen/fuzzyopen.py", line 77, in run
    self.itemsReady.emit(results)
RuntimeError: wrapped C/C++ object of type ScannerThread has been deleted

and Enki's window disappeared.

Started this session again. Pressed Ctrl+M. "loading" -> "many". Looks like a sad story.

---

Looks like it depends on current directory. Trying this feature when Enki is started from it's source directory works as described. Opening the same file, but with current directory=$HOME => problems (I see it scanning quarter-of-million-files compressed mount and other things accessible from $HOME. Maybe it should not follow mount points or symlinks?).

Also the similar fuzzy logic can apply to identifier completion (including cross-file).

---

Pressing Esc while it is still "loading" emits QThread: Destroyed while thread is still running in console.

---

Restarted Enki in a project directory with 8742 files (restoring session, but now with "correct" cwd). Ctrl+M works as it should, but is slow. There are also a lot of files (with particular extensions or in particular directories) I'd like to exlcude. Can it follow .gitignores?

[andreikop]

Thanks for the feedback about the crash and freeze. I'm going to optimize it for large directories, and respect .gitignore. This version is a proof of concept to discuss UI and project support.

What do you think about Project and merging navigation dialogs?

[andreikop]

@vi, with fuzzy search you'll probably don't need quick search in opened files. Just use regular file open functionality. Attempt to open opened file will switch focus to it. If necessary, I can tune best-match selection algorithm to prefer already opened files.

[vi]

I think the usual Locator should stay the same and not try to scan multiple directories or apply heuristics (at least now). There can be one-time hint like "Check the new feature" when using the Locator the first time in new Enki version.

The FuzzyOpener can instead be more like locator:

• Understand ../ as "please scan upper directory as well;
• *.exe as "please include *.exe files now, I really want to open one";
• f<space> as strongly prefer current file's directory for now;
• l<space> maybe should just duplicate Locator logic for jumping to lines for now. Maybe l qqq should offer the list of lines containing qqq to jump (in current file, maybe in other files as secondary choice);

I don't feel FuzzyOpener should be that dependent on Enki process's cwd. My ideas of what directory to scan instead should follow.

In any case, the directory (directories) being scanned should be shown, especially when it's "loading".

[vi]

Another bug:

I typed LME in FuzzyLocator, it showed the correct file. I continued: LME 23 meaning "Open 23'th line of this file". The file disappeared. I pressed Enter:

Traceback (most recent call last):
  File "/mnt/src/git/enki/bin/../enki/plugins/fuzzyopen/fuzzyopen.py", line 180, in _onEnter
    path = self._model.path(index)
  File "/mnt/src/git/enki/bin/../enki/plugins/fuzzyopen/fuzzyopen.py", line 125, in path
    return self._items[index.row()][0]
IndexError: list index out of range

[vi]

One more issue: excess case sensivity. I type A.mk. It correctly guess Android.mk. I try a.mk. It offers (among other things) Application.mk. Android.mk is low on the list and is only because of upper directory has a in the name.

I think it should also match case-insensitively, but with lower weight.

---

One more issue: when there are only 0-1 characters typed, there are too many items and it gets slow.

I think in this case it should only show N most popular items.

[bjones1]

Wow. I really like this. It's fantastic. I'm also a strong supporter of a project and project root; I'd like to require users to explicitly change the root, rather than accidentally navigating away from it in the file browser (oops! Didn't mean to press enter on that subdirectory!). I also find the locator of limited use, especially in light of the fuzzy opener. So, I would suggest:

1. Use the file browser's directory as the project root (already done). Disable navigating away from it; instead, add a [...] button on the combo box, allowing users to pick a new root using a select directory dialog.
2. Replace the Locator with the fuzzy opener. The locator's functions are available on hotkeys already: Ctrl+G to goto line, Ctrl+Alt+S to save as, and fuzzy awesomeness instead of f. Vi's suggestion of would also work nicely.

[vi]

@bjones1, Awesomeness is good, but stability is important too. I don't want breaking my already learned Locator ways. Unlike fuzzy opener, it is fast and deterministic.

Maybe when the fuzzy opener is polished and completely superseds Locator (including in performance), the latter can be abandoned. But I expect there to be multiple stable versions of Enki with both FuzzyOpener and Locator available simultaneously. The can be new functions to Locator even when FuzzyOpener has been stabilized. And FuzzyOpener should not scan and remember any directories if user never press Ctrl+M or whatever, i.e. don't use FuzzyLocator (i.e FuzzyOpener) => don't pay for it.

I don't use Enki's file browser, so not sure how it's directory is managed. But I expect directories where more than one file is opened to be available for FuzzyLocator too (with some weight).

---

Another feature request: showing FuzzyOpener content even before everything is scanned + scan directories in complicated, weighted order (important first, supplementary second).

For example, if I have files opened:

/home/vi/code/myproject1/src/1.cpp
/home/vi/code/myproject1/src/2.cpp
/home/vi/code/myproject1/src/3.c
/home/vi/code/myproject1/Makefile
/usr/share/doc/somelib/index.html
/usr/share/doc/somelib/types.html
/tmp/stacktrace.txt

Then I expect /home/vi/code/myproject1/src and *.cpp files to have maximum weight and scanned first. But /usr/share/doc/somelib/ may also appear (scanned later and skipped if it is too big, i.e. low prio).

As FuzzyLocator has "opened a can of worms of nondeterminism", more and more fuzzy logic can flow in: deciding what to show, what to scan (and in what order), which colour to display, which directories, which extensions, how to sort, whether to peek into content, how to parse user's input, how to consider accessed/modified date or not, what is too big, what is too small, how to analyse previous user's input, file popularity, etc. (but too far: using Internet because of FuzzyLocator is too much).

---

And, obviously, if user types explicit path to existing file /absulute/path/to/my/file.txt, it should show and open that file regardless of scanned or not directory, ignored or not filename extension and so on.

[andreikop]

@vi, amazing productivity! I created a milestone from your comments. Will process one by one. You probably need to subscribe to every issue to receive further notifications.

[andreikop]

It seems like @vi uses Locator, and @bjones uses File Browser. I use one for long jumps and the other for short.

I very like current deterministic Locator and would like to save this functionality. What do you think about updating Locator with Fuzzy opener functionality, so it will be Locator v2? Default behavior will be fuzzy-search. But current commands will be available. Current f command will work like now - no any fuzzy logic, autocompletion, no directory scanning. UI will look like current Fuzzy Opener. Ctrl+L shortcut is fine.

?

[andreikop]

Suggestions about how to implement project support are welcome!

Fixed root for file browser seems reasonable. But, it is probably not enough. @vi doesn't use File Browser, but I hope to make him so happy with project, that he will not use named sessions.

[vi]

that he will not use named sessions.

Currently I maintain 2-level hierarchy for Enki: Enki windows ( = sessions) -> Opened files. It is like what I do with consoles (Gnome-terminal windows -> tabs) or did with browser (Firefox windows -> tabs). As I often keep a lot of (somethings hundreds) objects (opened files in Enki, tabs in browser or console) alive, just single level hierarchy is not enough.

In browser I abandoned multi-window approach by using TreeStyleTabs (makes tabs like a collapsible tree, not just flat list). So for me to abandon sessions, I need:

1. Enki showing multiple windows or having opened files organized in a collapsible tree;
2. Ability to specify in which area (window, project, tree node) the remote open will be performed;
3. Ability to call Enki from console to show particular window (or expand particular tree);
4. FuzzyOpener should know what project (window or tree node) I am working on and suggest files from it. Other project's files may be accessible with lower weight and opening them may cause optional switching to other project.

Basically it is like multiple sessions emulated within a single sesion. I'm not sure if it is a good architecture and anyway it is not going to happen soon.

[vi]

Ctrl+L shortcut for both Locator and FuzzyOpener

Maybe it can be OK, but in this case unless I start typing something that is not a number, f ..., l ... and so on it should not begin to scan the files.

Actually like this approach:

In Settings -> Application shortcuts -> Navigations there are both Locator and FuzzyOpen (oh, it is so right now). If they are assigned to separate shortcuts (like Ctrl+L and Ctrl+M) then FuzzyOpener can begin it's fuzziness (i.e. scanning files and showing results) immediately. But if they are assigned to the same shortcut (both to Ctrl+L) then special merged mode should trigger and fuzziness should kick when user enters something that is not a Locator command.

Or there can be separate explicit configuration settings "merge FuzzyOpener and Locator" that removes both "Locator" and "Fuzzy open" items and adds "Fuzzy locator" item instead that works in a mode described above.

If separate handling is not easy to implement then maybe Locator and FuzzyOpen should remain separate for more time.

[vi]

You probably need to subscribe to every issue to receive further notifications.

Happens automatically: "You’re receiving notifications because you were mentioned."

[andreikop]

@vi, since you don't use file browser, I added p Locator command for you to open projects.

[andreikop]

@bjones1, @vi, current fuzzyopen branch works fine for me. I'm going to fix existing tests, write tests for fuzzyopen, merge the branch to master and release it. Maybe you can test the branch and say what do you think about it?

Git support and sessions connected to project path are big functionality. Leaving for the next release.

[vi]

Should some message be shown to user that receive this upgrade without expecting a major-ish change?

The message may emphasise:

• Current directory when starting Enki now matters;
• Enki may scan large amount of files;
• "Ctrl+L, newfile.c" is now "Ctrl+L, ./newfile.c";

[andreikop]

When some software tries to show tip of the day or release notes or something else, I usually immediately close the dialog and feel angry. I added more help to Locator start page about exact file open command. When FS is scanned progress is always visible on Locator dialog. I'll also write a post on Enki front page. And probably a Habr post.

[andreikop]

Forget fuzzyopen branch, now the code in the master

[andreikop]

First version works fine for me. Will be released soon