Port to the `notmuch2` bindings

[lenormf]

I migrated the code to use the notmuch2 bindings. Keep in mind:

• I haven't tested the code beyond running setup.py test
• named queries are not supported (no sure how they're handled by the new bindings)
• I don't send emails much from Alot (there might be dead paths in my execution flow)
• more commits will be pushed on this branch as I fix bugs

I'll run this for some time to fix things I might have missed. Ultimately, the commit history is not very relevant and can be squashed, when more testing has been done.

[pazz]

To fix travis, you want to change this line I believe..

[lenormf]

Yea, I'll go over the errors and warnings soon to fix them.

[lenormf]

I fixed the Travis manifest, but now I'm waiting on a NotMuch2 bindings package (PyPi) to update the setuptools dependency requirements.

[pazz]

Sorry but what has this got to do with pypi? aren't you installing the bindings locally from git master on travis?

[lenormf]

setup.py requires Pypi packages, which makes Travis fail, as far as I understand.

I've been using this branch locally with notmuch2 bindings since I created the PR, no problems so far.

[pazz]

Which ones? Could you point me to the logs?

I see that travis fails because something isn't implemented:

Traceback (most recent call last):
1347  File "/home/travis/build/pazz/alot/tests/db/test_manager.py", line 49, in test_save_named_query
1348    self.manager.flush()
1349  File "/home/travis/build/pazz/alot/alot/db/manager.py", line 110, in flush
1350    raise DatabaseError("unimplemented")
1351alot.db.errors.DatabaseError: unimplemented

[pazz]

I just tried this locally and it seems to work nicely so far. The travis build fails because the tests fail! (even locally for me), so no surprise at all. It seems before this can go ahead we need that the notmuch2 bindings implement the getter and setter for notmuch config options, because that is what the "saved search" feature is based on (both in emacs and alot).

[lenormf]

There's this.

As said in the original message, named queries and configuration option storage doesn't seem to be implemented, hence the exception. I'll wait for an official release of the new bindings to look into it more.

[pazz]

Ah I see. This is likely due to setup.py still containing a dependency to notmuch>=0.27.

[paretje]

When reading an unread mail using notmuch 0.30~rc1, I get:

DEBUG:thread:Tbuffer: auto remove unread tag from msg?
DEBUG:thread:Tbuffer: removing unread
DEBUG:manager:write-out item: ('untag', <function Message.remove_tags.<locals>.myafterwards at 0x7fb5ccd985e0>, 'id:qutebrowser/qutebrowser/issues/3636/640864575@github.com', ['unread'])
DEBUG:manager:cmd created
DEBUG:manager:got write lock
DEBUG:manager:got atomic
DEBUG:manager:ended atomic
DEBUG:manager:closed db
DEBUG:manager:<function Message.remove_tags.<locals>.myafterwards at 0x7fb5ccd985e0>
ERROR:ui:Traceback (most recent call last):
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 723, in apply_command
    cmd.apply(self)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/commands/globals.py", line 611, in apply
    ui.dbman.flush()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/db/manager.py", line 146, in flush
    afterwards()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/db/message.py", line 223, in myafterwards
    self._tags = self._tags.difference(tags)
AttributeError: 'MutableTagSet' object has no attribute 'difference'

self._tags - tags should work though.

[lenormf]

Thanks for the report!

I have a local fix for that bug, still testing it before I push it.

I fixed it by converting the tags cache to a set explicitly, instead of using whatever is returned by the bindings (a collections.abc.Set that doesn't have union/difference/etc.).

[paretje]

__isub__ is implemented though (in case of collections.abc.MutableSet).

Another issue I've had a couple of times now is:

DEBUG:thread:Tbuffer: auto remove unread tag from msg?
DEBUG:thread:Tbuffer: No, mid locked for autorm-unread
INFO:globals:refocussing
DEBUG:thread:Tbuffer: auto remove unread tag from msg?
DEBUG:thread:Tbuffer: No, mid locked for autorm-unread
DEBUG:ui:Got key (['q'], [113])
DEBUG:ui:cmdline: 'bclose'
DEBUG:ui:thread command string: "bclose"
DEBUG:__init__:mode:thread got commandline "bclose"
DEBUG:__init__:ARGS: ['bclose']
DEBUG:__init__:cmd parms {'redraw': None, 'force': False}
INFO:ui:closing current buffer [thread] Subject (1 message)
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
DEBUG:utils:unquoted header: |sender|
ERROR:ui:Traceback (most recent call last):
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 721, in apply_command
    await cmd.apply(self)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/commands/globals.py", line 484, in apply
    ui.buffer_close(self.buffer, self.redraw)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 447, in buffer_close
    self.buffer_focus(nextbuffer, redraw)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 477, in buffer_focus
    self.update()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 668, in update
    self.mainloop.draw_screen()
  File "/usr/lib/python3/dist-packages/urwid/main_loop.py", line 586, in draw_screen
    canvas = self._topmost_widget.render(self.screen_size, focus=True)
  File "/usr/lib/python3/dist-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/decoration.py", line 226, in render
    canv = self._original_widget.render(size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/container.py", line 1085, in render
    body = self.body.render((maxcol, maxrow-ftrim-htrim),
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/buffers/buffer.py", line 19, in render
    return self.body.render(size, focus)
  File "/usr/lib/python3/dist-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/listbox.py", line 470, in render
    middle, top, bottom = self.calculate_visible(
  File "/usr/lib/python3/dist-packages/urwid/listbox.py", line 353, in calculate_visible
    self._set_focus_complete( (maxcol, maxrow), focus )
  File "/usr/lib/python3/dist-packages/urwid/listbox.py", line 734, in _set_focus_complete
    middle,top,bottom=self.calculate_visible((maxcol,maxrow),focus)
  File "/usr/lib/python3/dist-packages/urwid/listbox.py", line 416, in calculate_visible
    next, pos = self._body.get_next( pos )
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/walker.py", line 46, in get_next
    return self._get_at_pos(start_from + self.direction)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/walker.py", line 72, in _get_at_pos
    widget = self._get_next_item()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/walker.py", line 85, in _get_next_item
    next_widget = self.containerclass(next_obj, **self.kwargs)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/widgets/search.py", line 26, in __init__
    self.rebuild()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/widgets/search.py", line 32, in rebuild
    self.thread = self.dbman.get_thread(self.tid)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/db/manager.py", line 257, in get_thread
    return Thread(self, self._get_notmuch_thread(tid))
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/db/manager.py", line 248, in _get_notmuch_thread
    db = Database(path=self.path, mode=Database.MODE.READ_ONLY)
  File "/usr/lib/python3/dist-packages/notmuch2/_database.py", line 129, in __init__
    raise errors.NotmuchError(ret, msg)
notmuch2.XapianError: A Xapian exception occurred opening database: Error opening table '/home/kevin/.mail/.notmuch/xapian/record.':
Couldn't open /home/kevin/.mail/.notmuch/xapian/record.baseA: Too many open files
Couldn't open /home/kevin/.mail/.notmuch/xapian/record.baseB: Too many open files

ERROR:ui:Traceback (most recent call last):
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 721, in apply_command
    await cmd.apply(self)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/commands/globals.py", line 484, in apply
    ui.buffer_close(self.buffer, self.redraw)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 447, in buffer_close
    self.buffer_focus(nextbuffer, redraw)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 477, in buffer_focus
    self.update()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 668, in update
    self.mainloop.draw_screen()
  File "/usr/lib/python3/dist-packages/urwid/main_loop.py", line 586, in draw_screen
    canvas = self._topmost_widget.render(self.screen_size, focus=True)
  File "/usr/lib/python3/dist-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/decoration.py", line 226, in render
    canv = self._original_widget.render(size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/container.py", line 1085, in render
    body = self.body.render((maxcol, maxrow-ftrim-htrim),
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/buffers/buffer.py", line 19, in render
    return self.body.render(size, focus)
  File "/usr/lib/python3/dist-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/listbox.py", line 470, in render
    middle, top, bottom = self.calculate_visible(
  File "/usr/lib/python3/dist-packages/urwid/listbox.py", line 353, in calculate_visible
    self._set_focus_complete( (maxcol, maxrow), focus )
  File "/usr/lib/python3/dist-packages/urwid/listbox.py", line 734, in _set_focus_complete
    middle,top,bottom=self.calculate_visible((maxcol,maxrow),focus)
  File "/usr/lib/python3/dist-packages/urwid/listbox.py", line 416, in calculate_visible
    next, pos = self._body.get_next( pos )
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/walker.py", line 46, in get_next
    return self._get_at_pos(start_from + self.direction)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/walker.py", line 72, in _get_at_pos
    widget = self._get_next_item()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/walker.py", line 85, in _get_next_item
    next_widget = self.containerclass(next_obj, **self.kwargs)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/widgets/search.py", line 26, in __init__
    self.rebuild()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/widgets/search.py", line 32, in rebuild
    self.thread = self.dbman.get_thread(self.tid)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/db/manager.py", line 257, in get_thread
    return Thread(self, self._get_notmuch_thread(tid))
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/db/manager.py", line 248, in _get_notmuch_thread
    db = Database(path=self.path, mode=Database.MODE.READ_ONLY)
  File "/usr/lib/python3/dist-packages/notmuch2/_database.py", line 129, in __init__
    raise errors.NotmuchError(ret, msg)
notmuch2.XapianError: A Xapian exception occurred opening database: Error opening table '/home/kevin/.mail/.notmuch/xapian/record.':
Couldn't open /home/kevin/.mail/.notmuch/xapian/record.baseA: Too many open files
Couldn't open /home/kevin/.mail/.notmuch/xapian/record.baseB: Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 277, in apply_commandline
    await apply_this_command(c)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 725, in apply_command
    self._error_handler(e)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 160, in _error_handler
    self.notify(msg, priority='error')
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 624, in notify
    self.update()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 660, in update
    lines.append(self.build_statusbar())
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 681, in build_statusbar
    info['total_messages'] = self.dbman.count_messages('*')
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/db/manager.py", line 236, in count_messages
    db = Database(path=self.path, mode=Database.MODE.READ_ONLY)
  File "/usr/lib/python3/dist-packages/notmuch2/_database.py", line 129, in __init__
    raise errors.NotmuchError(ret, msg)
notmuch2.XapianError: A Xapian exception occurred opening database: Error opening table '/home/kevin/.mail/.notmuch/xapian/record.':
Couldn't open /home/kevin/.mail/.notmuch/xapian/record.baseA: Too many open files
Couldn't open /home/kevin/.mail/.notmuch/xapian/record.baseB: Too many open files

ERROR:base_events:Task exception was never retrieved
future: <Task finished name='Task-76' coro=<UI._input_filter.<locals>._apply_fire() done, defined at /home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py:193> exception=XapianError()>
Traceback (most recent call last):
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 721, in apply_command
    await cmd.apply(self)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/commands/globals.py", line 484, in apply
    ui.buffer_close(self.buffer, self.redraw)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 447, in buffer_close
    self.buffer_focus(nextbuffer, redraw)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 477, in buffer_focus
    self.update()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 668, in update
    self.mainloop.draw_screen()
  File "/usr/lib/python3/dist-packages/urwid/main_loop.py", line 586, in draw_screen
    canvas = self._topmost_widget.render(self.screen_size, focus=True)
  File "/usr/lib/python3/dist-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/decoration.py", line 226, in render
    canv = self._original_widget.render(size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/container.py", line 1085, in render
    body = self.body.render((maxcol, maxrow-ftrim-htrim),
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/buffers/buffer.py", line 19, in render
    return self.body.render(size, focus)
  File "/usr/lib/python3/dist-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/listbox.py", line 470, in render
    middle, top, bottom = self.calculate_visible(
  File "/usr/lib/python3/dist-packages/urwid/listbox.py", line 353, in calculate_visible
    self._set_focus_complete( (maxcol, maxrow), focus )
  File "/usr/lib/python3/dist-packages/urwid/listbox.py", line 734, in _set_focus_complete
    middle,top,bottom=self.calculate_visible((maxcol,maxrow),focus)
  File "/usr/lib/python3/dist-packages/urwid/listbox.py", line 416, in calculate_visible
    next, pos = self._body.get_next( pos )
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/walker.py", line 46, in get_next
    return self._get_at_pos(start_from + self.direction)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/walker.py", line 72, in _get_at_pos
    widget = self._get_next_item()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/walker.py", line 85, in _get_next_item
    next_widget = self.containerclass(next_obj, **self.kwargs)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/widgets/search.py", line 26, in __init__
    self.rebuild()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/widgets/search.py", line 32, in rebuild
    self.thread = self.dbman.get_thread(self.tid)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/db/manager.py", line 257, in get_thread
    return Thread(self, self._get_notmuch_thread(tid))
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/db/manager.py", line 248, in _get_notmuch_thread
    db = Database(path=self.path, mode=Database.MODE.READ_ONLY)
  File "/usr/lib/python3/dist-packages/notmuch2/_database.py", line 129, in __init__
    raise errors.NotmuchError(ret, msg)
notmuch2.XapianError: A Xapian exception occurred opening database: Error opening table '/home/kevin/.mail/.notmuch/xapian/record.':
Couldn't open /home/kevin/.mail/.notmuch/xapian/record.baseA: Too many open files
Couldn't open /home/kevin/.mail/.notmuch/xapian/record.baseB: Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 277, in apply_commandline
    await apply_this_command(c)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 725, in apply_command
    self._error_handler(e)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 160, in _error_handler
    self.notify(msg, priority='error')
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 624, in notify
    self.update()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 660, in update
    lines.append(self.build_statusbar())
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 681, in build_statusbar
    info['total_messages'] = self.dbman.count_messages('*')
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/db/manager.py", line 236, in count_messages
    db = Database(path=self.path, mode=Database.MODE.READ_ONLY)
  File "/usr/lib/python3/dist-packages/notmuch2/_database.py", line 129, in __init__
    raise errors.NotmuchError(ret, msg)
notmuch2.XapianError: A Xapian exception occurred opening database: Error opening table '/home/kevin/.mail/.notmuch/xapian/record.':
Couldn't open /home/kevin/.mail/.notmuch/xapian/record.baseA: Too many open files
Couldn't open /home/kevin/.mail/.notmuch/xapian/record.baseB: Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 195, in _apply_fire
    await self.apply_commandline(cmdline)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 283, in apply_commandline
    self._error_handler(e)
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 160, in _error_handler
    self.notify(msg, priority='error')
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 624, in notify
    self.update()
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 660, in update
    lines.append(self.build_statusbar())
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/ui.py", line 681, in build_statusbar
    info['total_messages'] = self.dbman.count_messages('*')
  File "/home/kevin/.local/lib/python3.8/site-packages/alot-0.9.1-py3.8.egg/alot/db/manager.py", line 236, in count_messages
    db = Database(path=self.path, mode=Database.MODE.READ_ONLY)
  File "/usr/lib/python3/dist-packages/notmuch2/_database.py", line 129, in __init__
    raise errors.NotmuchError(ret, msg)
notmuch2.XapianError: A Xapian exception occurred opening database: Error opening table '/home/kevin/.mail/.notmuch/xapian/record.':
Couldn't open /home/kevin/.mail/.notmuch/xapian/record.baseA: Too many open files
Couldn't open /home/kevin/.mail/.notmuch/xapian/record.baseB: Too many open files

[pazz]

I wonder if these may better reported on the notmuch list so that the person responsible for the notmuch2 bindings can fix these bugs there?

[paretje]

Well, I'm now able to reproduce the last issue reliably by opening a couple of search buffers and switching rapidly between them. If I just set the initial_command, it always crashes. But to report this to notmuch, I guess we need a way to reproduce this with a minimal example using the bindings?

initial_command = "search tag:inbox; search tag:spam; bnext; bnext"

[lenormf]

I pushed a commit that should hopefully fix issues with tags.

I couldn't reproduce the other issues you mentioned, even with the given reproducer.

I'm a little concerned by these "unquoted header" errors, it looks like an infinite loop that makes Xapian exhausts the pool of available file descriptors.

[lenormf]

I can confirm either Alot and/or NotMuch are leaking file descriptors.

$ watch -n 1 ls -1 /proc/**pid**/fd/ | wc -l

Open threads, expand them all, you'll see the file descriptor count increase rapidly.

[lenormf]

If you need a stupid hotfix, the following dummy patch should prevent file descriptors from leaking:

diff --git a/alot/db/manager.py b/alot/db/manager.py
index 11bcd021..4d93324e 100644
--- a/alot/db/manager.py
+++ b/alot/db/manager.py
@@ -234,23 +234,29 @@ class DBManager:
     def count_messages(self, querystring):
         """returns number of messages that match `querystring`"""
         db = Database(path=self.path, mode=Database.MODE.READ_ONLY)
-        return db.count_messages(querystring,
+        ret = db.count_messages(querystring,
                                  exclude_tags=settings.get('exclude_tags'))
+        db.close()
+        return ret

     def count_threads(self, querystring):
         """returns number of threads that match `querystring`"""
         db = Database(path=self.path, mode=Database.MODE.READ_ONLY)
-        return db.count_threads(querystring,
+        ret = db.count_threads(querystring,
                                 exclude_tags=settings.get('exclude_tags'))
+        db.close()
+        return ret

     def _get_notmuch_thread(self, tid):
         """returns :class:`notmuch2.Thread` with given id"""
         db = Database(path=self.path, mode=Database.MODE.READ_ONLY)
         try:
-            return next(db.threads('thread:' + tid))
+            ret = next(db.threads('thread:' + tid))
         except NotmuchError:
             errmsg = 'no thread with id %s exists!' % tid
             raise NonexistantObjectError(errmsg)
+        db.close()
+        return ret

     def get_thread(self, tid):
         """returns :class:`Thread` with given thread id (str)"""
@@ -261,10 +267,12 @@ class DBManager:
         mode = Database.MODE.READ_ONLY
         db = Database(path=self.path, mode=mode)
         try:
-            return db.find_message(mid)
+            ret = db.find_message(mid)
         except:
             errmsg = 'no message with id %s exists!' % mid
             raise NonexistantObjectError(errmsg)
+        db.close()
+        return ret

     def get_message(self, mid):
         """returns :class:`Message` with given message id (str)"""
@@ -276,7 +284,9 @@ class DBManager:
         :rtype: list of str
         """
         db = Database(path=self.path)
-        return [t for t in db.tags]
+        ret = [t for t in db.tags]
+        db.close()
+        return ret

     def get_named_queries(self):
         """
@@ -284,7 +294,9 @@ class DBManager:
         :rtype: dict (str -> str) mapping alias to full query string
         """
         db = Database(path=self.path)
-        return {k[6:]: db.config[k] for k in db.config if k.startswith('query.')}
+        ret = {k[6:]: db.config[k] for k in db.config if k.startswith('query.')}
+        db.close()
+        return ret

     def get_threads(self, querystring, sort='newest_first', exclude_tags=None):
         """
@@ -309,6 +321,7 @@ class DBManager:
                             sort=self._sort_orders[sort],
                             exclude_tags=settings.get('exclude_tags')):
             yield t.threadid
+        db.close()

     def add_message(self, path, tags=None, afterwards=None):
         """

I think the database is only closed automatically when used in a context (with … as …:). I will ask the NotMuch devs if that's by design, or if I have to properly fix my code.

https://github.com/notmuch/notmuch/blob/93cc4b99dff2dea1f4af3b39f1864eb976c37648/bindings/python-cffi/notmuch2/_database.py#L256-L257

[paretje]

Thanks, I'll try that later today. About the unquoted headers: those are just the various senders of mails in that searchbuffer. I should have made that more clear, sorry.

[lenormf]

No problem, that's good to hear!

[lenormf]

I've had a bunch of problems cascading one after the other in a domino effect, I'll post a message once I've fixed them all.

[flub]

Hi! I'm the original author of the notmuch2 code. Someone pointed me to this PR to check out what troubles there are for you using it. There are certainly known gaps, I avoided implementing things that had no users yet whatsoever in order not to get it completely wrong. Also there will be bugs. So I would be grateful if you could post to the notmuch mailing list whenever you discover an issue or something missing. I'm not always super responsive but will normally try and read all emails to the list with "python" in them eventually, though feel free to CC flub@devork.be directly if I seem to have missed something. Also feel free to ping me here directly with questions.

[flub]

I fixed it by converting the tags cache to a set explicitly, instead of using whatever is returned by the bindings (a collections.abc.Set that doesn't have union/difference/etc.).

I've posted a patch to the notmuch list which adds these missing methods. I had no idea collections.abc.Set omits them!

[flub]

I think the database is only closed automatically when used in a context (with … as …:). I will ask the NotMuch devs if that's by design, or if I have to properly fix my code.

While it is true I tried to encourage using as a with statement, I'm not entirely sure what is going wrong here. The bindings call notmuch_database_destroy on __del__ which afaik does not require one to first call notmuch_database_close. So especially with CPython's reference counting I would not expect it to leak filedescriptors so rapidly.

I'd be interested if this still happens if you add self.close() to the Database.__del__ method? If that fixes it maybe it's a bug in notmuch itself?

[lenormf]

Hi @flub, thanks for the help! I subscribed to your ML.

I dropped the commit that converts NotMuch tag sets into Python sets, so far so good.

I tried calling self.close() before self._destroy(), it unfortunately didn't fix the leak. Here's a list of all the file descriptors open by the Alot process, after opening a few threads:

lrwx------ 1 fle users 64 Jun 16 10:47 0 -> /dev/pts/9
lrwx------ 1 fle users 64 Jun 16 10:47 1 -> /dev/pts/9
l-wx------ 1 fle users 64 Jun 16 10:47 10 -> 'pipe:[27008346]'
lr-x------ 1 fle users 64 Jun 16 10:47 11 -> 'pipe:[27008347]'
l-wx------ 1 fle users 64 Jun 16 10:47 12 -> 'pipe:[27008347]'
lr-x------ 1 fle users 64 Jun 16 10:47 13 -> /home/fle/mail/.notmuch/xapian/docdata.glass
lr-x------ 1 fle users 64 Jun 16 10:47 14 -> /home/fle/mail/.notmuch/xapian/termlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 15 -> /home/fle/mail/.notmuch/xapian/position.glass
lr-x------ 1 fle users 64 Jun 16 10:47 16 -> /home/fle/mail/.notmuch/xapian/postlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 17 -> /home/fle/mail/.notmuch/xapian/docdata.glass
lr-x------ 1 fle users 64 Jun 16 10:47 18 -> /home/fle/mail/.notmuch/xapian/termlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 19 -> /home/fle/mail/.notmuch/xapian/position.glass
lrwx------ 1 fle users 64 Jun 16 10:47 2 -> /dev/pts/9
lr-x------ 1 fle users 64 Jun 16 10:47 20 -> /home/fle/mail/.notmuch/xapian/postlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 21 -> /home/fle/mail/.notmuch/xapian/docdata.glass
lr-x------ 1 fle users 64 Jun 16 10:47 22 -> /home/fle/mail/.notmuch/xapian/termlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 23 -> /home/fle/mail/.notmuch/xapian/position.glass
lr-x------ 1 fle users 64 Jun 16 10:47 24 -> /home/fle/mail/.notmuch/xapian/postlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 25 -> /home/fle/mail/.notmuch/xapian/docdata.glass
lr-x------ 1 fle users 64 Jun 16 10:47 26 -> /home/fle/mail/.notmuch/xapian/termlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 27 -> /home/fle/mail/.notmuch/xapian/position.glass
lr-x------ 1 fle users 64 Jun 16 10:47 28 -> /home/fle/mail/.notmuch/xapian/postlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 29 -> /home/fle/mail/.notmuch/xapian/docdata.glass
lrwx------ 1 fle users 64 Jun 16 10:47 3 -> 'anon_inode:[eventpoll]'
lr-x------ 1 fle users 64 Jun 16 10:47 30 -> /home/fle/mail/.notmuch/xapian/termlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 31 -> /home/fle/mail/.notmuch/xapian/position.glass
lr-x------ 1 fle users 64 Jun 16 10:47 32 -> /home/fle/mail/.notmuch/xapian/postlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 33 -> /home/fle/mail/.notmuch/xapian/docdata.glass
lr-x------ 1 fle users 64 Jun 16 10:47 34 -> /home/fle/mail/.notmuch/xapian/termlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 35 -> /home/fle/mail/.notmuch/xapian/position.glass
lr-x------ 1 fle users 64 Jun 16 10:47 36 -> /home/fle/mail/.notmuch/xapian/postlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 37 -> /home/fle/mail/.notmuch/xapian/docdata.glass
lr-x------ 1 fle users 64 Jun 16 10:47 38 -> /home/fle/mail/.notmuch/xapian/termlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 39 -> /home/fle/mail/.notmuch/xapian/position.glass
lrwx------ 1 fle users 64 Jun 16 10:47 4 -> 'socket:[27008343]'
lr-x------ 1 fle users 64 Jun 16 10:47 40 -> /home/fle/mail/.notmuch/xapian/postlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 41 -> /home/fle/mail/.notmuch/xapian/docdata.glass
lr-x------ 1 fle users 64 Jun 16 10:47 42 -> /home/fle/mail/.notmuch/xapian/termlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 43 -> /home/fle/mail/.notmuch/xapian/position.glass
lr-x------ 1 fle users 64 Jun 16 10:47 44 -> /home/fle/mail/.notmuch/xapian/postlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 45 -> /home/fle/mail/.notmuch/xapian/docdata.glass
lr-x------ 1 fle users 64 Jun 16 10:47 46 -> /home/fle/mail/.notmuch/xapian/termlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 47 -> /home/fle/mail/.notmuch/xapian/position.glass
lr-x------ 1 fle users 64 Jun 16 10:47 48 -> /home/fle/mail/.notmuch/xapian/postlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 49 -> /home/fle/mail/.notmuch/xapian/docdata.glass
lrwx------ 1 fle users 64 Jun 16 10:47 5 -> 'socket:[27008344]'
lr-x------ 1 fle users 64 Jun 16 10:47 50 -> /home/fle/mail/.notmuch/xapian/termlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 51 -> /home/fle/mail/.notmuch/xapian/position.glass
lr-x------ 1 fle users 64 Jun 16 10:47 52 -> /home/fle/mail/.notmuch/xapian/postlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 57 -> /home/fle/mail/.notmuch/xapian/docdata.glass
lr-x------ 1 fle users 64 Jun 16 10:47 58 -> /home/fle/mail/.notmuch/xapian/termlist.glass
lr-x------ 1 fle users 64 Jun 16 10:47 59 -> /home/fle/mail/.notmuch/xapian/position.glass
lr-x------ 1 fle users 64 Jun 16 10:47 6 -> 'pipe:[27008345]'
lr-x------ 1 fle users 64 Jun 16 10:47 60 -> /home/fle/mail/.notmuch/xapian/postlist.glass
l-wx------ 1 fle users 64 Jun 16 10:47 7 -> 'pipe:[27008345]'
l-wx------ 1 fle users 64 Jun 16 10:47 8 -> /dev/null
lr-x------ 1 fle users 64 Jun 16 10:47 9 -> 'pipe:[27008346]'

I also verified that the current upstream version of Alot didn't leak file descriptors (it doesn't). Nothing changed from the original API to the new API, and my branch doesn't change how Alot creates databases connections (i.e. a Database() object is created, and the garbage collector handles the rest with no explicit call to close()).

So I assumed that Alot had just been lucky the entire time (the file descriptors were somehow closed), and that we had to use a context to Do Things Properly©. The problem with the new bindings is that after closing the database, iterators to data (for example, threads) become subsequently invalid. Alot relies on data being still valid after the Database() object used to recover it has been destroyed.

[lenormf]

Notes regarding the Travis tests:

• the notmuch Pypi package requires Python>=3.7, so the 3.6 environment fails - it's not clear whether that requirement is justified

• the tests for Python 3.7 and 3.8 fail because of a missing symbol, which was implemented recently - it might be a caching issue, as the commit that implements the symbol should be present in the Git repository pulled before the tests are run

• we have to wait until a Pypi package for the notmuch2 bindings is made available to satisfy Alot's dependencies in setup.py

• it might be simpler to make Travis use Debian (if possible), so we can pull the experimental python3-notmuch2 package maintained by the NotMuch devs directly

• Travis is extremely slow and cumbersome, I've had a great time with Cirrus - it's fast, flexible, easy… maybe the project should use that instead of tweaking .travis.yml do death

[pazz]

Quoting Frank LENORMAND (2020-06-17 08:58:15)

Notes regarding the Travis tests:

• the notmuch Pypi package requires Python>=3.7, so the 3.6 environment fails

• it's not clear whether that requirement is justified

  • the tests for Python 3.7 and 3.8 fail because of a missing symbol, which was implemented recently - it might be a caching issue, as the commit that implements the symbol should be present in the Git repository pulled before the tests are run

I'm not sure which symbol you are talking about but it this refers to a change in notmuch then note that our travis build installs notmuch and its bindings directly from git master https://github.com/pazz/alot/blob/master/.travis.yml#L50

• we have to wait until a Pypi package for the notmuch2 bindings is made available to satisfy Alot's dependencies in setup.py

I don't think we need to have anything to do with pypi. As mentioned above, the travis build gets notmuch from git master. The setup.py requirements do not have to be fulfilled by pypi, and are OK once your local environment provides them, which we do on Travis.

• it might be simpler to make Travis use Debian (if possible), so we can pull the experimental python3-notmuch2 package maintained by the NotMuch devs directly

• Travis is extremely slow and cumbersome, I've had a great time with Cirrus

• it's fast, flexible, easy… maybe the project should use that instead of tweaking .travis.yml do death

I agree that travis is not aging well, and changing its config file in the repo messes up our commit history. Now that github has their own CI system I wonder if it is not best to move to that in the long run. I am not convinced that moving to yet another third party CI system will be that advantageous (compared to github actions, which seem straightforward), but I can be convinced if need be :)

[lenormf]

I mentioned the upstream Git repository is cloned, and that the commit that implements Database.config is there already, that's why I also mentioned caching.

notmuch in setup.py refers to the original mappings, which this PR deprecates, so the line in question needs to point at the FFI bindings.

[pazz]

Quoting Frank LENORMAND (2020-06-17 09:34:48)

I mentioned the upstream Git repository is cloned, and that the commit that implements Database.config is there already, that's why I also mentioned caching.

I see. thanks.

notmuch in setup.py refers to the original mappings, which this PR deprecates, so the line in question needs to point at the FFI bindings.

Agreed.

[flub]

While on a walk today I thought of why the filedescriptors are likely leaking with notmuch2. I think this crossed my mind before but I dismissed it for some reason. Anyway, when you do something like this:

def func():
    db = notmuch2.Database(...)
    msg = db.find(...)
    return msg

The msg object still has a reference to the database, in order for the memory model to remain safe. This keeps the database object alive and it's __del__() is not called until msg is deleted. If however you do:

def func():
    notmuch2.Database(...) as db:
        msg = db.find(...)
    return msg

Now the db.close() has been called by the context manager. The msg object is still keeping the db object alive and that is why it hasn't been destroyed yet, but the database is now closed.

I think this is different from the orignal notmuch bindings which did not keep objects alive by reference and thus the db.__del__() would have been called much sooner in the first case.

I don't think there are circular references which would keep things alive forever in the former case. If there are that could also be a bug you stumbled across. I'll see if I can add some testing for this as well. And generally increase testing with operations on a closed database etc.

[lenormf]

I don't use the with context manager construct any more since the objects obtained with it become invalid.

Is there anything I can do to prevent leaks, with the old way?

[pazz]

With notmuch 30 released, this PR is getting ever more important. Could you comment on the status?

[ff2000]

With notmuch 30 released, this PR is getting ever more important. Could you comment on the status?

0.31 is out now.

@flub IMO fixed the FD leakage in https://github.com/lenormf/alot/pull/1 @lenormf can you confirm that works for you, as well?

With that PR alot seems to work for me like it did with the old bindings. At least the features I use didn't show any bad behaviour.

[pazz]

@lenormf could you add @ff2000 's patch to this PR ? It then remains to fix travis and this PR can be merged into alot master..

[ff2000]

@lenormf could you add @ff2000 's patch to this PR ? It then remains to fix travis and this PR can be merged into alot master..

@pazz the PR (and code) over at https://github.com/lenormf/alot/pull/1 is from @flub ;) (Yes, it's getting confusing, hehe)

[pazz]

I've moved this and Flub's commit into a new PR (#1547) and merged that into the master branch with some small pep8 fixes. The Travis builds look fine (although in the long run I guess it makes sense to clean this up a bit).

Thanks for all your work on this! On my own I would not have had the time and energy to move alot to notmuch2!