Unable to process files

[ydzhou]

vim 8.2 with latest ctrlpvim/ctrlp running on mac os 10.12.3

Getting following errors when run Ctrlp

function ctrlp#init[31]..<SNR>84_setlines_post[6]..ctrlp#files[8]..<SNR>84_GlobPath[13]..<SNR>84_GlobPath[13]..<SNR
E484: Cannot open file  /Users/xxx/Library/Application Support/MobileSync
E484: Cannot open file /Users/xxx/Library/Application Support/CallHistoryTransactions
E484: Cannot open file /Users/xxx/Library/Application Support/com.apple.sharedfilelist

I do not run into this issue if I change to fd for user command, or if I use kien/ctrlp.vim

[ydzhou]

Looks like it's the same issue with https://github.com/ctrlpvim/ctrlp.vim/issues/587

[ydzhou]

This part of code is causing issue https://github.com/ctrlpvim/ctrlp.vim/blob/02d72d6a4008b0f0ef28a3bbefa67c231645fcaa/autoload/ctrlp.vim#L429

[mattn]

Please paste full error-messages. Part of stacktrace are trimed.

[ydzhou]

Sure.

Error detected while processing function ctrlp#init[31]..<SNR>78_setlines_post[6]..ctrlp#files[8]..<SNR>78_GlobPath[13]..<SNR>78_GlobPath[13]..<SNR>78_GlobPath:
E484: Can't open file /Users/xxx/Library/Application Support/MobileSync
E484: Can't open file /Users/xxx/Library/Application Support/CallHistoryTransactions
E484: Can't open file /Users/xxx/Library/Application Support/com.apple.sharedfilelist
E484: Can't open file /Users/xxx/Library/Application Support/Knowledge
E484: Can't open file /Users/xxx/Library/Application Support/com.apple.TCC
E484: Can't open file /Users/xxx/Library/Application Support/FileProvider
E484: Can't open file /Users/xxx/Library/Application Support/CallHistoryDB
E484: Can't open file /Users/xxx/Library/Assistant/SiriVocabulary
E484: Can't open file /Users/xxx/Library/Group Containers/group.com.apple.secure-control-center-preferences
E484: Can't open file /Users/xxx/Library/Group Containers/group.com.apple.notes
E484: Can't open file /Users/xxx/Library/Containers/com.apple.VoiceMemos
E484: Can't open file /Users/xxx/Library/Containers/com.apple.archiveutility

[mattn]

Hmm, can't repro. Do you set maxfuncdepth ?

[mattn]

Which patch level version of Vim do you use?

[Shougo]

The files permissions are readable?

[ydzhou]

Hmm, can't repro. Do you set maxfuncdepth ?

No. I did not touch that part.

[ydzhou]

The files permissions are readable?

Yes

[ydzhou]

output of vim --version

VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Mar 20 2022 17:46:06)
macOS version - arm64
Included Patches: 1-4600
Compiled by Homebrew

+acl               +file_in_path      +mouse_urxvt       -tag_any_white
+arabic            +find_in_path      +mouse_xterm       -tcl
+autocmd           +float             +multi_byte        +termguicolors
+autochdir         +folding           +multi_lang        +terminal
-autoservername    -footer            -mzscheme          +terminfo
-balloon_eval      +fork()            +netbeans_intg     +termresponse
+balloon_eval_term +gettext           +num64             +textobjects
-browse            -hangul_input      +packages          +textprop
++builtin_terms    +iconv             +path_extra        +timers
+byte_offset       +insert_expand     +perl              +title
+channel           +ipv6              +persistent_undo   -toolbar
+cindent           +job               +popupwin          +user_commands
-clientserver      +jumplist          +postscript        +vartabs
+clipboard         +keymap            +printer           +vertsplit
+cmdline_compl     +lambda            +profile           +vim9script
+cmdline_hist      +langmap           -python            +viminfo
+cmdline_info      +libcall           +python3           +virtualedit
+comments          +linebreak         +quickfix          +visual
+conceal           +lispindent        +reltime           +visualextra
+cryptv            +listcmds          +rightleft         +vreplace
+cscope            +localmap          +ruby              +wildignore
+cursorbind        +lua               +scrollbind        +wildmenu
+cursorshape       +menu              +signs             +windows
+dialog_con        +mksession         +smartindent       +writebackup
+diff              +modify_fname      -sodium            -X11
+digraphs          +mouse             -sound             -xfontset
-dnd               -mouseshape        +spell             -xim
-ebcdic            +mouse_dec         +startuptime       -xpm
+emacs_tags        -mouse_gpm         +statusline        -xsmp
+eval              -mouse_jsbterm     -sun_workshop      -xterm_clipboard
+ex_extra          +mouse_netterm     +syntax            -xterm_save
+extra_search      +mouse_sgr         +tag_binary        
-farsi             -mouse_sysmouse    -tag_old_static    

[Shougo]

Hm. If so, it seems too many open files error in macOS. macOS file descriptor limit is very few in default.

Ah, it seems same with previous issue. Because the directory contains spaces. So the issue should be closed.

[ydzhou]

Hm. If so, it seems too many open files error in macOS. macOS file descriptor limit is very few in default.

I am not sure if it is an issue of operating system itself, because kien/ctrlp.vim works and if I comment out that section, it also works.

[ydzhou]

The minimal vimrc I use and issue still persists.

call plug#begin()
" Bundles
Plug 'ctrlpvim/ctrlp.vim'
call plug#end()

set nocompatible
set backspace=indent,eol,start
set noeb vb t_vb=
if has('autocmd')
        autocmd GUIEnter * set visualbell t_vb=
endif
set encoding=UTF-8

[Shougo]

Hm. I will look the issue.

[Shougo]

Please test the case. I need the reproduce steps.

https://github.com/ctrlpvim/ctrlp.vim/issues/587#issuecomment-1000722243

[Shougo]

I cannot reproduce your errors.

[ydzhou]

Please test the case. I need the reproduce steps.

#587 (comment)

Yes I can reproduce the issue with this test case.

[mattn]

Could you please confirm your ctrlpvim/ctrlp.vim is latest? This change must be included.

https://github.com/ctrlpvim/ctrlp.vim/commit/352b628d0ac18a17c6e4fed2f5c5b0b2c69b7970

[Shougo]

Really? I think your ctrlp.vim is not the latest. I cannot reproduce the problem in the case.

[ydzhou]

xxx @ xxx-iMac • 22-03-21 • ~/dotfiles/vim/plugged/ctrlp.vim > git log
commit 02d72d6a4008b0f0ef28a3bbefa67c231645fcaa (grafted, HEAD -> master, origin/master, origin/HEAD)
Author: mattn <mattn.jp@gmail.com>
Date:   Thu Mar 17 02:43:56 2022 +0900

    Merge pull request #597 from ctrlpvim/optimize5

    Lines can be omitted over s:maxlines

[mattn]

Could you please try this?

:echo readdir('/Users/xxx/Library/Application Support/com.apple.sharedfilelist', '1', {'sort': 'none'})

[ydzhou]

Your change looks like not merged? https://github.com/ctrlpvim/ctrlp.vim/blob/master/autoload/ctrlp.vim#L432

[ydzhou]

Could you please try this?

:echo readdir('/Users/xxx/Library/Application Support/com.apple.sharedfilelist', '1', {'sort': 'none'})

It throws the same error.

[mattn]

Well, the path /Users/xxx/Library/Application Support/com.apple.sharedfilelist exists?

[ydzhou]

Well, the path /Users/xxx/Library/Application Support/com.apple.sharedfilelist exists?

Yeah it exists.

[ydzhou]

Let's taking a step back. I actually put those configs for ctrlp.vim previously. But why is it still trying to search in my Library folder?

let g:ctrlp_map = '<leader>/'
let g:ctrlp_working_path_mode = 'ra'
set wildignore+=*/Downloads/*,*/Applications/*,*/Movies/*,*Pictures/*,*/Music/*,*/Library/*,*/tmp/*,*.so,*.o,*.a,*.obj,*.swp,*.zip,*.pyc,*.pyo,*.class,.DS_Store
let g:ctrlp_custom_ignore = '\.git$\|\.hg$\|\.svn$\|\.rvm$'
let g:ctrlp_max_height=30

[Shougo]

I think wildignore is not used in ctrlp.

[ydzhou]

The files permissions are readable?

Very weird. The file permission is drwxr-xr-x. But when I try ls, I get Operation not permitted. So I was wrong on the read permission. I think this is related to the permission issue.

[mattn]

Could you please try this patch?

diff --git a/autoload/ctrlp.vim b/autoload/ctrlp.vim
index fcb1823..260eb64 100644
--- a/autoload/ctrlp.vim
+++ b/autoload/ctrlp.vim
@@ -430,9 +430,12 @@ if has('patch-8.2-0995')
        let entries = []
        let dirs = substitute(a:dirs, '\\\([%# ]\)', '\1', 'g')
        for e in split(dirs, ',')
-           sil let files = readdir(e, '1', {'sort': 'none'})
-           if !s:showhidden | cal filter(files, 'v:val[0] != "."') | en
-           let entries += map(files, 'e.s:lash.v:val')
+           try
+               let files = readdir(e, '1', {'sort': 'none'})
+               if !s:showhidden | cal filter(files, 'v:val[0] != "."') | en
+               let entries += map(files, 'e.s:lash.v:val')
+           cat
+           endt
        endfo
        let [dnf, depth] = [ctrlp#dirnfile(entries), a:depth + 1]
        if &wig != '' | cal filter(dnf[1], 'glob(v:val) != ""') | en

[ydzhou]

Interesting. wildignore was working in kien/ctrlp.vim. But after I fixed the ignore, there is no more issue. It can also open file with space in the name.

[ydzhou]

Could you please try this patch?

diff --git a/autoload/ctrlp.vim b/autoload/ctrlp.vim
index fcb1823..260eb64 100644
--- a/autoload/ctrlp.vim
+++ b/autoload/ctrlp.vim
@@ -430,9 +430,12 @@ if has('patch-8.2-0995')
      let entries = []
      let dirs = substitute(a:dirs, '\\\([%# ]\)', '\1', 'g')
      for e in split(dirs, ',')
-         sil let files = readdir(e, '1', {'sort': 'none'})
-         if !s:showhidden | cal filter(files, 'v:val[0] != "."') | en
-         let entries += map(files, 'e.s:lash.v:val')
+         try
+             let files = readdir(e, '1', {'sort': 'none'})
+             if !s:showhidden | cal filter(files, 'v:val[0] != "."') | en
+             let entries += map(files, 'e.s:lash.v:val')
+         cat
+         endt
      endfo
      let [dnf, depth] = [ctrlp#dirnfile(entries), a:depth + 1]
      if &wig != '' | cal filter(dnf[1], 'glob(v:val) != ""') | en

Yeah no error... but I am not seeing those two folders in the list. Is it expected?

[mattn]

kien/ctrlp.vim had the problem of being very slow when dealing with large files. ctrlpvim/ctrlp.vim includes many improvements for faster speeds.

[ydzhou]

kien/ctrlp.vim had the problem of being very slow when dealing with large files. ctrlpvim/ctrlp.vim incorporates many improvements for faster speeds.

Yeah. I am getting too used to kien's version and making some false assumption on how ctrlpvim/ctrlp.vim will work.

Thank @mattn and @Shougo 's time to resolve this issue! Have a good day.

[mattn]

How about this changes?

diff --git a/autoload/ctrlp.vim b/autoload/ctrlp.vim
index fcb1823..855e2c0 100644
--- a/autoload/ctrlp.vim
+++ b/autoload/ctrlp.vim
@@ -100,6 +100,7 @@ let [s:pref, s:bpref, s:opts, s:new_opts, s:lc_opts] =
    \ 'bufpath_mod':           ['s:bufpath_mod', ':~:.:h'],
    \ 'formatline_func':       ['s:flfunc', 's:formatline(v:val)'],
    \ 'user_command_async':    ['s:usrcmdasync', 0],
+   \ 'use_readdir':           ['s:usereaddir', 1],
    \ }, {
    \ 'open_multiple_files':   's:opmul',
    \ 'regexp':                's:regexp',
@@ -425,14 +426,17 @@ fu! s:CloseCustomFuncs()
    en
 endf

-if has('patch-8.2-0995')
+if has('patch-8.2-0995') && s:usereaddir
    fu! s:GlobPath(dirs, depth)
        let entries = []
        let dirs = substitute(a:dirs, '\\\([%# ]\)', '\1', 'g')
        for e in split(dirs, ',')
-           sil let files = readdir(e, '1', {'sort': 'none'})
-           if !s:showhidden | cal filter(files, 'v:val[0] != "."') | en
-           let entries += map(files, 'e.s:lash.v:val')
+           try
+               let files = readdir(e, '1', {'sort': 'none'})
+               if !s:showhidden | cal filter(files, 'v:val[0] != "."') | en
+               let entries += map(files, 'e.s:lash.v:val')
+           cat
+           endt
        endfo
        let [dnf, depth] = [ctrlp#dirnfile(entries), a:depth + 1]
        if &wig != '' | cal filter(dnf[1], 'glob(v:val) != ""') | en

If you set let g:ctrlp_use_readdir = 0, ctrlpvim/ctrlp.vim will use older GlobPath.

[ydzhou]

How about this changes?

diff --git a/autoload/ctrlp.vim b/autoload/ctrlp.vim
index fcb1823..855e2c0 100644
--- a/autoload/ctrlp.vim
+++ b/autoload/ctrlp.vim
@@ -100,6 +100,7 @@ let [s:pref, s:bpref, s:opts, s:new_opts, s:lc_opts] =
  \ 'bufpath_mod':           ['s:bufpath_mod', ':~:.:h'],
  \ 'formatline_func':       ['s:flfunc', 's:formatline(v:val)'],
  \ 'user_command_async':    ['s:usrcmdasync', 0],
+ \ 'use_readdir':           ['s:usereaddir', 1],
  \ }, {
  \ 'open_multiple_files':   's:opmul',
  \ 'regexp':                's:regexp',
@@ -425,14 +426,17 @@ fu! s:CloseCustomFuncs()
  en
 endf

-if has('patch-8.2-0995')
+if has('patch-8.2-0995') && s:usereaddir
  fu! s:GlobPath(dirs, depth)
      let entries = []
      let dirs = substitute(a:dirs, '\\\([%# ]\)', '\1', 'g')
      for e in split(dirs, ',')
-         sil let files = readdir(e, '1', {'sort': 'none'})
-         if !s:showhidden | cal filter(files, 'v:val[0] != "."') | en
-         let entries += map(files, 'e.s:lash.v:val')
+         try
+             let files = readdir(e, '1', {'sort': 'none'})
+             if !s:showhidden | cal filter(files, 'v:val[0] != "."') | en
+             let entries += map(files, 'e.s:lash.v:val')
+         cat
+         endt
      endfo
      let [dnf, depth] = [ctrlp#dirnfile(entries), a:depth + 1]
      if &wig != '' | cal filter(dnf[1], 'glob(v:val) != ""') | en

If you set let g:ctrlp_use_readdir = 0, ctrlpvim/ctrlp.vim will use older GlobPath.

Sounds good

[ydzhou]

One suggestion is that even CtrlP is not able to open those files for some reason, can we try catch the error per file so that CtrlP still list files that are processed successfully but discard ones with error gracefully?

[mattn]

The patch above I wrote is ignoring for each errors that could not reaed the files.

[mattn]

Ignore erorrs and added that option.

https://github.com/ctrlpvim/ctrlp.vim/commit/3f1ba868bda7fc7d34eb7829a5248094c6a41b10

[ydzhou]

I open a PR to update the documentation for wildignore. Can you take a look?

[mattn]

Merged. Thank you. Closable?

[ydzhou]

Yes, please.

Thank you!