OSError: Too many open files (OSX)

[rgorel]

Given:

• MBP Early 2015
• OS X El Capitan 10.11.1 (15B42)
• rails project with 9061 files
• Sublime Text 3 build 3114
• SublimeFileBrowser v. 0.7.11

Steps to reproduce:

1. Open FileBrowser
2. Browse through a couple of directories inside the project

Problem: Multiple error messages with the text 'Too many open files' start popping up.

Stacktraces:

error: Too many open files
error: Too many open files
Exception in thread Thread-29:
Traceback (most recent call last):
  File "./python3.3/threading.py", line 901, in _bootstrap_inner
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/api.py", line 146, in run
    self.queue_events(self.timeout)
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/kqueue.py", line 686, in queue_events
    self.watch.is_recursive)
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/utils/dirsnapshot.py", line 236, in __init__
    for p, st in walk(path):
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/utils/dirsnapshot.py", line 213, in walk
    paths = [os.path.join(root, name) for name in listdir(root)]
OSError: [Errno 24] Too many open files: '/Users/gorerom/dev/index-service'

Traceback (most recent call last):
  File "/Applications/Sublime Text.app/Contents/MacOS/sublime_plugin.py", line 553, in run_
    return self.run()
  File "git_gutter in /Users/gorerom/Library/Application Support/Sublime Text 3/Installed Packages/GitGutter.sublime-package", line 34, in run
  File "view_collection in /Users/gorerom/Library/Application Support/Sublime Text 3/Installed Packages/GitGutter.sublime-package", line 59, in untracked
  File "git_gutter_handler in /Users/gorerom/Library/Application Support/Sublime Text 3/Installed Packages/GitGutter.sublime-package", line 266, in untracked
  File "git_gutter_handler in /Users/gorerom/Library/Application Support/Sublime Text 3/Installed Packages/GitGutter.sublime-package", line 282, in handle_files
  File "git_gutter_handler in /Users/gorerom/Library/Application Support/Sublime Text 3/Installed Packages/GitGutter.sublime-package", line 347, in run_command
  File "./python3.3/subprocess.py", line 784, in __init__
  File "./python3.3/subprocess.py", line 1293, in _get_handles
OSError: [Errno 24] Too many open files
error: Too many open files
error: Too many open files
Exception in thread Thread-31:
Traceback (most recent call last):
  File "./python3.3/threading.py", line 901, in _bootstrap_inner
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/api.py", line 146, in run
    self.queue_events(self.timeout)
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/kqueue.py", line 686, in queue_events
    self.watch.is_recursive)
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/utils/dirsnapshot.py", line 236, in __init__
    for p, st in walk(path):
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/utils/dirsnapshot.py", line 213, in walk
    paths = [os.path.join(root, name) for name in listdir(root)]
OSError: [Errno 24] Too many open files: '/Users/gorerom/dev/index-service/app/models'

Traceback (most recent call last):
  File "/Applications/Sublime Text.app/Contents/MacOS/sublime_plugin.py", line 553, in run_
    return self.run()
  File "git_gutter in /Users/gorerom/Library/Application Support/Sublime Text 3/Installed Packages/GitGutter.sublime-package", line 34, in run
  File "view_collection in /Users/gorerom/Library/Application Support/Sublime Text 3/Installed Packages/GitGutter.sublime-package", line 59, in untracked
  File "git_gutter_handler in /Users/gorerom/Library/Application Support/Sublime Text 3/Installed Packages/GitGutter.sublime-package", line 266, in untracked
  File "git_gutter_handler in /Users/gorerom/Library/Application Support/Sublime Text 3/Installed Packages/GitGutter.sublime-package", line 282, in handle_files
  File "git_gutter_handler in /Users/gorerom/Library/Application Support/Sublime Text 3/Installed Packages/GitGutter.sublime-package", line 347, in run_command
  File "./python3.3/subprocess.py", line 784, in __init__
  File "./python3.3/subprocess.py", line 1293, in _get_handles
OSError: [Errno 24] Too many open files
Exception in thread Thread-30:
Traceback (most recent call last):
  File "./python3.3/threading.py", line 901, in _bootstrap_inner
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/api.py", line 146, in run
    self.queue_events(self.timeout)
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/kqueue.py", line 686, in queue_events
    self.watch.is_recursive)
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/utils/dirsnapshot.py", line 236, in __init__
    for p, st in walk(path):
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/utils/dirsnapshot.py", line 213, in walk
    paths = [os.path.join(root, name) for name in listdir(root)]
OSError: [Errno 24] Too many open files: '/Users/gorerom/dev/index-service/app'

Exception in thread Thread-32:
Traceback (most recent call last):
  File "./python3.3/threading.py", line 901, in _bootstrap_inner
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/api.py", line 146, in run
    self.queue_events(self.timeout)
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/kqueue.py", line 686, in queue_events
    self.watch.is_recursive)
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/utils/dirsnapshot.py", line 236, in __init__
    for p, st in walk(path):
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/utils/dirsnapshot.py", line 213, in walk
    paths = [os.path.join(root, name) for name in listdir(root)]
OSError: [Errno 24] Too many open files: '/Users/gorerom/dev/index-service/app/models/notification'

Exception in thread Thread-28:
Traceback (most recent call last):
  File "./python3.3/threading.py", line 901, in _bootstrap_inner
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/api.py", line 146, in run
    self.queue_events(self.timeout)
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/kqueue.py", line 686, in queue_events
    self.watch.is_recursive)
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/utils/dirsnapshot.py", line 236, in __init__
    for p, st in walk(path):
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/utils/dirsnapshot.py", line 213, in walk
    paths = [os.path.join(root, name) for name in listdir(root)]
OSError: [Errno 24] Too many open files: '/Users/gorerom/dev'

Exception in thread Thread-25:
Traceback (most recent call last):
  File "./python3.3/threading.py", line 901, in _bootstrap_inner
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/api.py", line 146, in run
    self.queue_events(self.timeout)
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/kqueue.py", line 686, in queue_events
    self.watch.is_recursive)
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/utils/dirsnapshot.py", line 236, in __init__
    for p, st in walk(path):
  File "/Users/gorerom/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/utils/dirsnapshot.py", line 213, in walk
    paths = [os.path.join(root, name) for name in listdir(root)]
OSError: [Errno 24] Too many open files: '/Users/gorerom'

I realize that tracebacks refer mostly to git gutter plugin, however, the issue reproduces only when using SublimeFileBrowser. I also know that I can raise the open files limit at OS level, but I'm afraid that it could lead to unexpected results / heavy resources consuming, also I doubt that it's necessary, because I have used SublimeFileBrowser along with GitGutter for many years and the issue has started appearing only from one of the recent sublime plugins update.

Please advise.

[vovkkk]

issue has started appearing only from one of the recent sublime plugins update

This is because the usage of watchdog module started only since 0.8.0, couple weeks ago.

First you can try install Python 3.3.3 (on your system), install watchdog (python3.3 pip install watchdog), find fsevents.so or something similar in directory where it was installed, e.g. /usr/local/lib/python3.3/site-packages/watchdog or whatever it is on your system, copy this file to Packages/watchdog/all/watchdog, restart ST, see if it helps; if it does, please, propose pull request to https://github.com/vovkkk/sublime-watchdog with this binary file.

Second (if first does not work or too complicated) you can disable auto-refresh globally or per view, it will prevent scheduling paths.

[vovkkk]

SublimeFileBrowser v. 0.7.11

Are you sure? it is probably 0.8.0

[rgorel]

So I had python 3.5 on my system and tried to install watchdog using it. I have not found a file named fsevents.so, but found observers/fsevents.py. I have replaced the one bundled with FileBrowser with this file and it did not help.

I haven't tried to install python 3.3.3.

Disabling auto-refresh fixed the issue.

[maximkoretskiy]

pip install don't not add .so files to installed package. So I had to:

• build watchdog from source https://github.com/gorakhargosh/watchdog#installation
• take _watchdog_fsevents.so from the root of installed package. Possible location variants:
  • /usr/local/Cellar/python33/3.3.6/lib/python3.3/site-packages/
  • /usr/local/lib/python3.3/site-packages
• put it to Packages/watchdog/all/watchdog

[maximkoretskiy]

@rgorel ^

[vovkkk]

@maximkoretskiy does it fix the issue? If yes, could you share _watchdog_fsevents.so file here or via pull request in https://github.com/vovkkk/sublime-watchdog

[maximkoretskiy]

Yes. Done.

Btw Great plugin!

[maximkoretskiy]

@vovkkk I think we should reopen this issue

[vovkkk]

Yes, due to incapability of Package Control we have to properly release it. I’ll do it in near future (tomorrow or day later)

[maximkoretskiy]

Just discovered that there is another problem. Sublime works longer till first error then before fix, but now I see in console such traces

Exception in thread Thread-723:
Traceback (most recent call last):
  File "./python3.3/threading.py", line 901, in _bootstrap_inner
  File "./python3.3/threading.py", line 858, in run
  File "/Users/remedy/Library/Application Support/Sublime Text 3/Packages/package_events/all/package_events.py", line 76, in _do_notify
    listener(package_name, event_name, payload)
  File "0_dired_fs_observer in /Users/remedy/Library/Application Support/Sublime Text 3/Installed Packages/FileBrowser.sublime-package", line 144, in dired_event_handler
  File "0_dired_fs_observer in /Users/remedy/Library/Application Support/Sublime Text 3/Installed Packages/FileBrowser.sublime-package", line 138, in <lambda>
  File "0_dired_fs_observer in /Users/remedy/Library/Application Support/Sublime Text 3/Installed Packages/FileBrowser.sublime-package", line 121, in finish_refresh
  File "/Users/remedy/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/api.py", line 353, in unschedule_all
    self._clear_emitters()
  File "/Users/remedy/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/api.py", line 230, in _clear_emitters
    for emitter in self._emitters:
ValueError: PyCapsule_GetPointer called with invalid PyCapsule object

[maximkoretskiy]

Anyway current behavior is better and it does not break another plugins/

[vovkkk]

Did you restart Sublime Text?

File "/Users/remedy/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/api.py", line 353, in unschedule_all self._clear_emitters() File "/Users/remedy/Library/Application Support/Sublime Text 3/Packages/watchdog/all/watchdog/observers/api.py", line 230, in _clear_emitters for emitter in self._emitters: ValueError: PyCapsule_GetPointer called with invalid PyCapsule object

This is upstream issue https://github.com/gorakhargosh/watchdog/issues/63

[maximkoretskiy]

Yes. Just restarted

[vovkkk]

Do you have two 0_dired_fs_observer.py files?

I mean judging by case in upstream issue, this exception happens when there are two observers.

So maybe you have Installed Packages/FileBrowser.sublime-package/0_dired_fs_observer.py and Packages/FileBrowser/0_dired_fs_observer.py or something like that? Because I’m pretty sure that normally SFB would create only one instance of observer.