search

isamert / scli

a simple terminal user interface for signal messenger (using signal-cli)
GNU General Public License v3.0
448 stars 39 forks source link

"Too many open files" when starting scli with 100s of unread group messages #199

Closed scottwn closed 1 year ago

scottwn commented 2 years ago

When I start scli while there are 100s of unread group messages in my account, startup fails with this error

Traceback (most recent call last):
  File "/Users/scott/scli/scli", line 5273, in <module>
    main()
  File "/Users/scott/scli/scli", line 5267, in main
    loop.run()
  File "/Users/scott/.pyenv/versions/3.10.0/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urwid/main_loop.py", line 287, in run
    self._run()
  File "/Users/scott/.pyenv/versions/3.10.0/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urwid/main_loop.py", line 385, in _run
    self.event_loop.run()
  File "/Users/scott/.pyenv/versions/3.10.0/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urwid/main_loop.py", line 790, in run
    self._loop()
  File "/Users/scott/.pyenv/versions/3.10.0/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urwid/main_loop.py", line 827, in _loop
    self._watch_files[fd]()
  File "/Users/scott/.pyenv/versions/3.10.0/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urwid/main_loop.py", line 233, in cb
    rval = callback(data)
  File "/Users/scott/scli/scli", line 656, in _daemon_stdout_handler
    self._envelope_handler(envelope)
  File "/Users/scott/scli/scli", line 693, in _envelope_handler
    self.callbacks['contacts_sync']()
  File "/Users/scott/scli/scli", line 4732, in _on_contacts_sync
    self._actions.reload()
  File "/Users/scott/scli/scli", line 4067, in reload
    self.update_groups_async()
  File "/Users/scott/scli/scli", line 4381, in update_groups_async
    self._daemon.get_group_members(
  File "/Users/scott/scli/scli", line 956, in get_group_members
    self._dbus_send_signal_cli(args, callback=proc_callback)
  File "/Users/scott/scli/scli", line 727, in _dbus_send_signal_cli
    return self._dbus_send(args, *proc_args, **proc_kwargs)
  File "/Users/scott/scli/scli", line 715, in _dbus_send
    proc = self.run(args, *proc_args, **proc_kwargs)
  File "/Users/scott/scli/scli", line 550, in run
    proc = subprocess.Popen(
  File "/Users/scott/.pyenv/versions/3.10.0/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 832, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "/Users/scott/.pyenv/versions/3.10.0/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1622, in _get_handles
    errread, errwrite = os.pipe()
OSError: [Errno 24] Too many open files

After the initial failure, I can start scli successfully as normal.

exquo commented 2 years ago

Looks like scli is starting too many processes in the background, opening a file handle for each. These are dbus-send commands that scli runs concurrently on startup, to query the signal-cli daemon for the user's groups' names and members.

Are you a member of many groups on this account? (I don't think the number of unread messages in those groups is a factor). I don't know why it works the second time; my guess is it's just luck, and that time the background processes return and close quickly enough to not hit the limit.

It's probably a good idea to limit the number of concurrent processes scli can spawn. In the meantime, you can increase the number of file handles any process can open (with e.g. ulimit -n 2048, this will only affect the current shell; see https://stackoverflow.com/questions/16526783/python-subprocess-too-many-open-files)

exquo commented 2 years ago

Also, scli has not been closing some of the file streams when it should have been. Could you please try the https://github.com/isamert/scli/tree/hotfix/199 branch and see if it fixes this issue?

scottwn commented 2 years ago

I'm not a member of too many groups. <10 groups.

I switched to hotfix/199 and since then I have not seen this issue, everything is working well.

garymm commented 1 year ago

I'm seeing this error on tip of master (e65ece04fbab2bbc79dce1724bc5b50d3888f8b3). Should I open another issue?

exquo commented 1 year ago

This is indeed still happening. (Especially after the changes in 8ccae9f08afd28682af69b9512304152a4e3b5af).

Tracking this issue in #205.