Improved performance

[mattn]

Anyone could you please try this?

globpath

readdir

[tacahiroy]

I have had a quick try for this and it appears that the readdir version is slower than the current implementation, master branch. Although I didn't investigate why, I took profile for s:GlobPath for both versions. Here are my set up:

• OS: CentOS 7.7
• Vim: 8.2.1520
• CtrlP: No variables set
• Project used: github.com/vim/vim

branch=master
for y in {1..5}; do
    # needed to press <Esc> by myself during indexing ...
    vim -c ':CtrlPClearCache' -c ":profile start /tmp/${master}${y}.txt" -c ':profile func *GlobPath*' -c ':CtrlP' -c ':quit'
done

master (d93d978)

Took around 8 sec to complete indexing the project.

$ cat /tmp/master1.txt
FUNCTION  <SNR>70_GlobPath()
    Defined: ~/dev/src/github.com/ctrlpvim/ctrlp.vim/autoload/ctrlp.vim:428
Called 7 times
Total time:   7.902779
 Self time:   2.037264

count  total (s)   self (s)
    7              0.437958     let entries = split(globpath(a:dirs, s:glob), "\n")
    7   2.040687   0.000261     let [dnf, depth] = [ctrlp#dirnfile(entries), a:depth + 1]
    7              0.001861     cal extend(g:ctrlp_allfiles, dnf[1])
    7   0.000329   0.000249     if !empty(dnf[0]) && !s:maxf(len(g:ctrlp_allfiles)) && depth <= s:maxdepth
    6   0.003648   0.000164             sil! cal ctrlp#progress(len(g:ctrlp_allfiles), 1)
    6   0.057730   0.004502             cal s:GlobPath(join(map(dnf[0], 's:fnesc(v:val, "g", ",")'), ','), depth)
    7              0.000025     en

FUNCTIONS SORTED ON TOTAL TIME
count  total (s)   self (s)  function
    7   7.902779   2.037264  <SNR>70_GlobPath()

FUNCTIONS SORTED ON SELF TIME
count  total (s)   self (s)  function
    7   7.902779   2.037264  <SNR>70_GlobPath()

readdir (1a8da4b)

Took around 10 sec to complete.

$ cat /tmp/readdir1.txt
FUNCTION  <SNR>70_GlobPath()
    Defined: ~/dev/src/github.com/ctrlpvim/ctrlp.vim/autoload/ctrlp.vim:429
Called 154 times
Total time:   9.980040
 Self time:   8.232291

count  total (s)   self (s)
  154              0.001602             let entries = []
  154   0.011079   0.002817             let slash = s:lash()
 3889              0.462273             for item in readdirex(a:dirs, '1', {'sort': 'none'})
 3735              0.039279                     let each = item['name']
 3735              0.034181                     let etype = item['type']
 3735              0.069383                     if !s:showhidden && each[0] == '.'
   11              0.000039                             con
 3724              0.013332                     en
 3724              0.051126                     let each = a:dirs.slash.each
 7294   1.661432   0.110584                     if s:igntype >= 0 && s:usrign(each, etype) | con | en
 3701              0.029833                     if etype == 'dir'
  153              0.004020                             cal extend(entries, s:GlobPath(each, a:depth + 1))
 3548              0.033043                     elsei etype == 'link' && s:folsym
                                                        let isfile = !isdirectory(each)
                                                        if s:folsym == 2 || !s:samerootsyml(each, isfile, cwd)
                                                                cal extend(entries, [each])
                                                        en
 3548              0.027115                     elsei etype == 'file'
 3548              0.062821                             cal extend(g:ctrlp_allfiles, [each])
 3548   0.147166   0.098697                             if !s:maxf(len(g:ctrlp_allfiles)) && a:depth <= s:maxdepth
 3548              0.046404                                     if len(g:ctrlp_allfiles) % 100 == 0
   35   0.015370   0.001000                                             sil! cal ctrlp#progress(len(g:ctrlp_allfiles), 1)
 3548              0.013386                                     en
 3548              0.055626                                     cal extend(entries, [each])
 3548              0.012887                             en
 3701              0.013677                     en
 3855              0.016480             endfor
  154              0.001075             retu entries

FUNCTIONS SORTED ON TOTAL TIME
count  total (s)   self (s)  function
  154   9.980040   8.232291  <SNR>70_GlobPath()

FUNCTIONS SORTED ON SELF TIME
count  total (s)   self (s)  function
  154   9.980040   8.232291  <SNR>70_GlobPath()

I'm wondering why I got the different result from your screen recording; I mean the readdir version is faster in your recordings. Do you have any clue why?

And please let me know if you need more information about my environment, set up, etc. Please let me know if you need some more information.

[mattn]

@tacahiroy Thanks your investigation

Do you set g:ctrlp_custom_ignore in your vimrc?

[tacahiroy]

Do you set g:ctrlp_custom_ignore in your vimrc?

No, I didn't set the variable when I was checking the readdir branch. Is it required to make it work?

[mattn]

Is it required to make it work?

No. I could confirm that the variable make ctrlp slower. Hmm

[mattn]

@tacahiroy Could you please try this?

    fu! s:GlobPath(dirs, depth)
        let entries = []
        for e in split(a:dirs, ',')
            sil let files = readdir(e, '1', {'sort': 'none'})
            if !s:showhidden | cal filter(files, 'v:val[0] != "."') | en
            let entries += map(files, 'e.s:lash.v:val')
        endfo
        let [dnf, depth] = [ctrlp#dirnfile(entries), a:depth + 1]
        cal extend(g:ctrlp_allfiles, dnf[1])
        if !empty(dnf[0]) && !s:maxf(len(g:ctrlp_allfiles)) && depth <= s:maxdepth
            sil! cal ctrlp#progress(len(g:ctrlp_allfiles), 1)
            cal s:GlobPath(join(map(dnf[0], 's:fnesc(v:val, "g", ",")'), ','), depth)
        en
    endf

[mattn]

master branch (d93d97813dc839ef0782302a0debd7c4877f09f3)

FUNCTION  <SNR>102_GlobPath()
    Defined: ~/.vim/plugged/ctrlp.vim/autoload/ctrlp.vim:428
Called 7 times
Total time:   3.255180
 Self time:   0.858566

count  total (s)   self (s)
    7              0.216790     let entries = split(globpath(a:dirs, s:glob), "\n")
    7   0.834405   0.000090     let [dnf, depth] = [ctrlp#dirnfile(entries), a:depth + 1]
    7              0.000414     cal extend(g:ctrlp_allfiles, dnf[1])
    7   0.000090   0.000070     if !empty(dnf[0]) && !s:maxf(len(g:ctrlp_allfiles)) && depth <= s:maxdepth
    6   0.012044   0.000056         sil! cal ctrlp#progress(len(g:ctrlp_allfiles), 1)
    6   0.012838   0.000830         cal s:GlobPath(join(map(dnf[0], 's:fnesc(v:val, "g", ",")'), ','), depth)
    7              0.000007     en

FUNCTIONS SORTED ON TOTAL TIME
count  total (s)   self (s)  function
    7   3.255180   0.858566  <SNR>102_GlobPath()

FUNCTIONS SORTED ON SELF TIME
count  total (s)   self (s)  function
    7   3.255180   0.858566  <SNR>102_GlobPath()

readdir branch (bdbde8d2774dc6e925836892d6cae2a94328a8d7)

FUNCTION  <SNR>102_GlobPath()
    Defined: ~/.vim/plugged/ctrlp.vim/autoload/ctrlp.vim:429
Called 7 times
Total time:   2.673139
 Self time:   0.605925

count  total (s)   self (s)
    7              0.000015         let entries = []
  161              0.000354         for e in split(a:dirs, ',')
  154              0.009378             sil let files = readdir(e, '1', {'sort': 'none'})
  448              0.011460             if !s:showhidden | cal filter(files, 'v:val[0] != "."') | en
  154              0.006960             let entries += map(files, 'e.s:lash.v:val')
  161              0.000156         endfo
    7   0.707577   0.000086         let [dnf, depth] = [ctrlp#dirnfile(entries), a:depth + 1]
    7              0.000520         cal extend(g:ctrlp_allfiles, dnf[1])
    7   0.000099   0.000076         if !empty(dnf[0]) && !s:maxf(len(g:ctrlp_allfiles)) && depth <= s:maxdepth
    6   0.128125   0.000144             sil! cal ctrlp#progress(len(g:ctrlp_allfiles), 1)
    6   0.012657   0.000864             cal s:GlobPath(join(map(dnf[0], 's:fnesc(v:val, "g", ",")'), ','), depth)
    7              0.000005         en

FUNCTIONS SORTED ON TOTAL TIME
count  total (s)   self (s)  function
    7   2.673139   0.605925  <SNR>102_GlobPath()

FUNCTIONS SORTED ON SELF TIME
count  total (s)   self (s)  function
    7   2.673139   0.605925  <SNR>102_GlobPath()

[mattn]

Using extend

FUNCTION  <SNR>102_GlobPath()
    Defined: ~/.vim/plugged/ctrlp.vim/autoload/ctrlp.vim:429
Called 7 times
Total time:   2.530760
 Self time:   0.651777

count  total (s)   self (s)
    7              0.000015         let entries = []
  161              0.000338         for e in split(a:dirs, ',')
  154              0.006642             sil let files = readdir(e, '1', {'sort': 'none'})
  448              0.063126             if !s:showhidden | cal filter(files, 'v:val[0] != "."') | en
  154              0.059255             cal extend(entries, map(files, 'e.s:lash.v:val'))
  161              0.000158         endfo
    7   0.648050   0.000087         let [dnf, depth] = [ctrlp#dirnfile(entries), a:depth + 1]
    7              0.000431         cal extend(g:ctrlp_allfiles, dnf[1])
    7   0.000115   0.000093         if !empty(dnf[0]) && !s:maxf(len(g:ctrlp_allfiles)) && depth <= s:maxdepth
    6   0.022147   0.000051             sil! cal ctrlp#progress(len(g:ctrlp_allfiles), 1)
    6   0.010645   0.000812             cal s:GlobPath(join(map(dnf[0], 's:fnesc(v:val, "g", ",")'), ','), depth)
    7              0.000006         en

FUNCTIONS SORTED ON TOTAL TIME
count  total (s)   self (s)  function
    7   2.530760   0.651777  <SNR>102_GlobPath()

FUNCTIONS SORTED ON SELF TIME
count  total (s)   self (s)  function
    7   2.530760   0.651777  <SNR>102_GlobPath()

[tacahiroy]

Great! 8857c17 improves much and is now faster than the current implementation on master branch.

• master (d93d978): 8.49551 sec
• readdir (8857c17): 7.33068 sec

[mattn]

Thank you!