Performance Question

[mbriggs]

Hi Le,

I have a question regarding perf. I am using flx through projectile on my projects files (~8k). When searching for something which contains many matches initially, flx will lock up for ~1s before it narrows to a point where it can display results.

What I am curious about is if I then backspace to nothing, and type in the same word, performance is smooth as silk. But if I dismiss flx and invoke it again, I see the same lock up. Is there anything I can do to cache whatever calculations flx is doing the first time through?

Even with that, flx is so much better then the alternatives I will live with it :)

[mbriggs]

Ok, digging into this a bit, it turns out that for large datasets, as you narrow down it is supposed to flip back to flx when it enters the threshold. Adding some debugging output to flx-ido-match shows me that it is getting passed the full set of candidates on each keystroke, so the threshold never gets entered.

[mbriggs]

I hacked up the way the choice was made here https://github.com/mbriggs/flx/commit/cc71f03f0f5b75135276451fdb5ca4e24b25ad3c which now works for me. Would make a pull request, but I really didnt do it in a terribly elegant way.

[lewang]

So the idea is in these cases, you would decrease the threshold until reaching a compromise of acceptable performance and match happiness. Obviously it's not working as it should I'll mark it as a bug and investigate further later.

[lewang]

I see the behaviour you describe. Definitely a bug.

[lewang]

Actually I recall in 24.4 a bunch of ido caching is coming in, so the behaviour has changed a lot.

[mbriggs]

using GNU Emacs 24.4.50.1 (x86_64-apple-darwin13.1.0, NS apple-appkit-1265.19) of 2014-04-01 on chillwind-yeti.local if that helps at all.

[lewang]

Can you try fix-ido-not-narrowing branch?

[mbriggs]

Amazing performance improvement! :+1:

One thing is that these changes cause a conflict with ido-hacks, which probably isnt a problem, but you may want to document somewhere. Performance is absolutely fantastic overall :D

[mbriggs]

one thing is if you backspace, flx matching is not invoked until you type again (or if it is, the faces aren't showing up). REALLY not a big deal though

[lewang]

one thing is if you backspace, flx matching is not invoked until you type again

Yeah I just realized this. I'll fix it soon (and ask you to test again).

On Wed, Apr 30, 2014 at 4:34 PM, Matt Briggs notifications@github.comwrote:

one thing is if you backspace, flx matching is not invoked until you type again (or if it is, the faces aren't showing up). REALLY not a big deal though

— Reply to this email directly or view it on GitHubhttps://github.com/lewang/flx/issues/55#issuecomment-41846747 .

Le

[lewang]

What exactly is the conflict with ido-hacks?

On Wed, Apr 30, 2014 at 4:23 PM, Matt Briggs notifications@github.comwrote:

Amazing performance improvement! [image: :+1:]

One thing is that these changes cause a conflict with ido-hacks, which probably isnt a problem, but you may want to document somewhere. Performance is absolutely fantastic overall :D

— Reply to this email directly or view it on GitHubhttps://github.com/lewang/flx/issues/55#issuecomment-41845597 .

Le

[mbriggs]

if ido-hacks is there and flex matching is enabled, no candidates show up at all if you are above the flx threshold.

[mbriggs]

i think the main reason I still had ido-hacks was that it enabled ido in a few places that even ido-ubiquitous doesnt. Given that the main point of the mode is to speed up flex, and flex speed seems to not be an issue at all on 24.4/with this branch, I would honestly rather just extract the bits that I want out of it then bringing the full library in anyways.

[mbriggs]

Ok, the fix in this branched works with most cases, but I was able to create a perf issue (pretty sure it is a corner case).

https://gist.github.com/mbriggs/65ab7c201e0739dce80c

That is a find . of a project, where there is a HUGE amount of files in a few directories. When I specifically narrow to one of those mega-directories, followed by trying to narrow further, flx will hang for a moment in the transition between flex and flx

An example of that (going against the list in the gist) would be layoutmaasvai. I am on a 2.8 ghz i7, have gc-cons-threshold set to 20000000, and flx-ido-threshold set to 2000.

[lewang]

Yes, I think I understand the problem. It's the same as flx not activating after backspacing.

On Wed, Apr 30, 2014 at 5:05 PM, Matt Briggs notifications@github.comwrote:

Ok, the fix in this branched works with most cases, but I was able to create a perf issue (pretty sure it is a corner case).

https://gist.github.com/mbriggs/65ab7c201e0739dce80c

That is a find . of a project, where there is a HUGE amount of files in a few directories. When I specifically narrow to one of those mega-directories, followed by trying to narrow further, flx will hang for a moment in the transition between flex and flx

An example of that (going against the list in the gist) would be layoutmaasvai. I am on a 2.8 ghz i7, have gc-cons-threshold set to 20000000, and flx-ido-threshold set to 2000.

— Reply to this email directly or view it on GitHubhttps://github.com/lewang/flx/issues/55#issuecomment-41850561 .

Le

[lewang]

Can you give this a go?

I've fixed a typo in the threshold variable name.

The default has also been lowered from 6000 to 500.

[mbriggs]

I think there is a case now where flx can get the wrong list of candidates, will try to find a reproducable set of steps

[lewang]

Have you seen this wrong candidates bug again?

On Thu, May 1, 2014 at 10:01 AM, Matt Briggs notifications@github.comwrote:

I think there is a case now where flx can get the wrong list of candidates, will try to find a reproducable set of steps

— Reply to this email directly or view it on GitHubhttps://github.com/lewang/flx/issues/55#issuecomment-41912152 .

Le

[mbriggs]

I havent... i hit two situations where i thought i should have seen results but didnt just during normal work. it honestly just could have been user error >.>

[lewang]

That's fine.

I'll probably cut a new version in a few days and update the README to recommend using flx as the "last mile" solution with a low flx-ido-threshold.

On Fri, May 2, 2014 at 11:23 AM, Matt Briggs notifications@github.comwrote:

I havent... i hit two situations where i thought i should have seen results but didnt just during normal work. it honestly just could have been user error >.>

— Reply to this email directly or view it on GitHubhttps://github.com/lewang/flx/issues/55#issuecomment-42043425 .

Le

[lewang]

@mbriggs Have you seen a repro lately?

[mbriggs]

nope, mostly sure it was user error :) been using https://github.com/topfunky/PeepOpen recently for finding project files though.

[lewang]

The fix has been merged into master