notmuch.errors.NullPointerError on query (tag:new)

[varac]

I'm using these versions, all installed as debian package from ubuntu bionic:

• afew 1.3.0-1,
• notmuch 0.26-1ubuntu3
• xapian 1.4.5-1

xapian-check returns no errors on the database.

This is my reduced afew config for reproducing the bug:

[global]                    

[SpamFilter]                
spam_tag = junk             

I get a this notmuch.errors.NullPointerError running afew:

» afew --tag --new -vv 
INFO:afew.filters.SpamFilter.SpamFilter:Tagging spam messages
DEBUG:root:Executing query '(tag:new)'
Traceback (most recent call last):
  File "/usr/bin/afew", line 11, in <module>
    load_entry_point('afew==1.3.0', 'console_scripts', 'afew')()
  File "/usr/lib/python3/dist-packages/afew/commands.py", line 159, in main
    inner_main(args, database, query_string)
  File "/usr/lib/python3/dist-packages/afew/main.py", line 23, in main
    filter_.run(query_string)
  File "/usr/lib/python3/dist-packages/afew/filters/BaseFilter.py", line 60, in run
    self.handle_message(message)
  File "/usr/lib/python3/dist-packages/afew/filters/HeaderMatchingFilter.py", line 28, in handle_message
    value = message.get_header(self.header)
  File "/usr/lib/python3/dist-packages/notmuch/message.py", line 234, in get_header
    raise NullPointerError()
notmuch.errors.NullPointerError

A notmuch query for tag:new runs successfully:

» notmuch search tag:new | wc -l
58176

Maybe afew is overwhelmed by the amount of new mails ? (I'm in the process of retagging)

[varac]

Trying a bit more I found that it's not only the spamfilter, but also the i.e. ListMailsFilter that breaks:

INFO:afew.filters.ListMailsFilter.ListMailsFilter:Tagging mailing list posts
DEBUG:root:Executing query '((tag:new)) AND (NOT tag:lists)'
Traceback (most recent call last):
  File "/usr/bin/afew", line 11, in <module>
    load_entry_point('afew==1.3.0', 'console_scripts', 'afew')()
  File "/usr/lib/python3/dist-packages/afew/commands.py", line 159, in main
    inner_main(args, database, query_string)
  File "/usr/lib/python3/dist-packages/afew/main.py", line 23, in main
    filter_.run(query_string)
  File "/usr/lib/python3/dist-packages/afew/filters/BaseFilter.py", line 60, in run
    self.handle_message(message)
  File "/usr/lib/python3/dist-packages/afew/filters/HeaderMatchingFilter.py", line 28, in handle_message
    value = message.get_header(self.header)
  File "/usr/lib/python3/dist-packages/notmuch/message.py", line 234, in get_header
    raise NullPointerError()
notmuch.errors.NullPointerError

[flokli]

That's an error from the python notmuch bindings.

Afew simply calls get_header from notmuch.Message, which returns "" if the header doesn't exist, or raises NullPointerError if an internal error occurs while talking to the notmuch library.

See lib/python3.6/site-packages/notmuch/message.py:

        #Returns NULL if any error occurs.
        header = Message._get_header(self._msg, _str(header))
        if header == None:
            raise NullPointerError()
        return header.decode('UTF-8', 'ignore')

So it's notmuch db that is in a limbo state, and nothing done wrong by afew. I suggest to recreate the database from scratch, while preserving tags via notmuch-dump and notmuch-restore.

Closing for now.

[varac]

@floki: I already got this after deleting the notmuch db, running notmuch new followed by the afew command above which broke. I can try again.

[dtzWill]

Ran into this today, using notmuch built from git. In my case, notmuch was running into limit on open file descriptors (1024). Not sure if it's a leak or a scope/lifetime change (I'd guess the former since no mention of the latter), and probably not anything to worry about from afew's perspective.

Seems that notmuch was opening file for each message (to grab the requested header), duplicating the file descriptor somewhere, and leaving one of them open. Ran into the problem while running HeaderMatchingFilter (as above). Shrug. Will bug upstream "soon" :).

Anyway, since this wasn't where I expected a "NullPointerError" to lead, sharing so perhaps it may help someone in the future :). And just saying it's not entirely inconsistent with what's reported here but my variant was definitely related to a bugged notmuch -- I and I'm sure many others have run afew on much more than 1024 messages at once, no issues there.

[flokli]

fixed via https://notmuchmail.org/pipermail/notmuch/2019/027971.html in https://git.notmuchmail.org/git?p=notmuch;a=commit;h=e19954fa188e8a2a9bb4d57a235eb11490880fb9