search

pazz / alot

Terminal-based Mail User Agent
GNU General Public License v3.0
696 stars 164 forks source link

search query slowness due to athor replacement #1520

Closed meskio closed 4 years ago

meskio commented 4 years ago

Describe the bug If thread_authors_replace_me is set every search buffer reload will need to read every single email from disk. This makes the search very slow mostly in the case of a lot of encrypted emails.

Software Versions

To Reproduce

  1. Set thread_authors_replace_me to true
  2. do a search including a lot of encrypted email. It will take some seconds to load while gpg is working.

I'm personally happy to keep thread_authors_replace_me to false in my setup, if I'm the only one affected by it we can close the issue.

paretje commented 4 years ago

I have the same issue. It doesn't need to be encrypted though, signed messages trigger this as well. Quite a lot of time is spent loading the email's from disk, and in gpg (getting the key and verifying signatures). None of this seems necessary, and is a regression from previous versions. I might have a look later on.

These are the top calls:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    472/1    0.003    0.000   40.620   40.620 {built-in method builtins.exec}
        1    0.000    0.000   40.620   40.620 <string>:1(<module>)
        1    0.000    0.000   40.620   40.620 /usr/lib/python3.8/runpy.py:197(run_module)
        1    0.000    0.000   40.619   40.619 /usr/lib/python3.8/runpy.py:64(_run_code)
        1    0.000    0.000   40.619   40.619 /home/kevin/vcs/once/alot/alot/__main__.py:4(<module>)
        1    0.000    0.000   40.188   40.188 /home/kevin/vcs/once/alot/alot/__main__.py:86(main)
        1    0.000    0.000   37.591   37.591 /home/kevin/vcs/once/alot/alot/ui.py:47(__init__)
        1    0.000    0.000   37.590   37.590 /usr/lib/python3/dist-packages/urwid/main_loop.py:276(run)
        1    0.000    0.000   37.590   37.590 /usr/lib/python3/dist-packages/urwid/main_loop.py:374(_run)
        1    0.000    0.000   37.589   37.589 /usr/lib/python3/dist-packages/urwid/main_loop.py:1328(run)
        1    0.000    0.000   37.589   37.589 /usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py:265(run)
        1    0.000    0.000   37.588   37.588 /usr/lib/python3.8/asyncio/base_events.py:557(run_forever)
       35    0.001    0.000   37.588    1.074 /usr/lib/python3.8/asyncio/base_events.py:1784(_run_once)
       44    0.000    0.000   26.248    0.597 /usr/lib/python3.8/asyncio/events.py:79(_run)
       44    0.000    0.000   26.248    0.597 {method 'run' of 'Context' objects}
       15    0.000    0.000   26.141    1.743 /home/kevin/vcs/once/alot/alot/ui.py:244(apply_commandline)
    17/15    0.000    0.000   26.133    1.742 /home/kevin/vcs/once/alot/alot/ui.py:704(apply_command)
       16    0.000    0.000   25.903    1.619 /usr/lib/python3/dist-packages/urwid/main_loop.py:574(draw_screen)
        6    0.000    0.000   25.890    4.315 /home/kevin/vcs/once/alot/alot/ui.py:646(update)
  6458/16    0.021    0.000   25.813    1.613 /usr/lib/python3/dist-packages/urwid/widget.py:138(cached_render)
  2716/11    0.007    0.000   25.813    2.347 /usr/lib/python3/dist-packages/urwid/decoration.py:219(render)
       11    0.000    0.000   25.787    2.344 /usr/lib/python3/dist-packages/urwid/container.py:1062(render)
       11    0.000    0.000   25.782    2.344 /home/kevin/vcs/once/alot/alot/buffers/buffer.py:18(render)
       10    0.001    0.000   25.779    2.578 /usr/lib/python3/dist-packages/urwid/listbox.py:462(render)
      326    0.002    0.000   25.461    0.078 /home/kevin/vcs/once/alot/alot/walker.py:78(_get_next_item)
    23/19    0.006    0.000   25.330    1.333 /usr/lib/python3/dist-packages/urwid/listbox.py:327(calculate_visible)
      326    0.001    0.000   25.277    0.078 /home/kevin/vcs/once/alot/alot/widgets/search.py:20(__init__)
      326    0.008    0.000   25.273    0.078 /home/kevin/vcs/once/alot/alot/widgets/search.py:31(rebuild)
     1568    0.002    0.000   25.263    0.016 /home/kevin/vcs/once/alot/alot/walker.py:61(_get_at_pos)
     1439    0.001    0.000   25.253    0.018 /home/kevin/vcs/once/alot/alot/walker.py:45(get_next)
     1304    0.009    0.000   24.766    0.019 /home/kevin/vcs/once/alot/alot/widgets/search.py:116(build_text_part)
     1304    0.005    0.000   24.688    0.019 /home/kevin/vcs/once/alot/alot/widgets/search.py:193(prepare_string)
      326    0.001    0.000   24.665    0.076 /home/kevin/vcs/once/alot/alot/widgets/search.py:177(prepare_authors_string)
      326    0.002    0.000   24.664    0.076 /home/kevin/vcs/once/alot/alot/db/thread.py:177(get_authors_string)
      326    0.006    0.000   24.648    0.076 /home/kevin/vcs/once/alot/alot/db/thread.py:148(get_authors)
      326    0.004    0.000   24.555    0.075 /home/kevin/vcs/once/alot/alot/db/thread.py:224(get_messages)
  991/436    0.015    0.000   24.326    0.056 /home/kevin/vcs/once/alot/alot/db/thread.py:236(accumulate)
      991    0.027    0.000   24.268    0.024 /home/kevin/vcs/once/alot/alot/db/message.py:29(__init__)
      991    0.023    0.000   23.353    0.024 /home/kevin/vcs/once/alot/alot/db/message.py:99(get_email)
      991    0.007    0.000   23.277    0.023 /home/kevin/vcs/once/alot/alot/db/utils.py:302(decrypted_message_from_bytes)
       14    0.000    0.000   21.436    1.531 /home/kevin/vcs/once/alot/alot/ui.py:193(_apply_fire)
        4    0.000    0.000   21.381    5.345 /home/kevin/vcs/once/alot/alot/commands/globals.py:170(apply)
        5    0.000    0.000   20.719    4.144 /usr/lib/python3/dist-packages/urwid/listbox.py:710(_set_focus_complete)
      991    0.011    0.000   16.882    0.017 /home/kevin/vcs/once/alot/alot/db/utils.py:251(_decrypted_message_from_message)
      230    0.047    0.000   16.110    0.070 /home/kevin/vcs/once/alot/alot/db/utils.py:98(_handle_signatures)
       35    0.001    0.000   11.338    0.324 /usr/lib/python3.8/selectors.py:451(select)
       35   11.337    0.324   11.337    0.324 {method 'poll' of 'select.epoll' objects}
      230    0.016    0.000   10.054    0.044 /home/kevin/vcs/once/alot/alot/db/utils.py:38(add_signature_headers)
      226    0.008    0.000    9.964    0.044 /home/kevin/vcs/once/alot/alot/crypto.py:29(get_key)
      226    0.012    0.000    8.585    0.038 /usr/lib/python3/dist-packages/gpg/core.py:1201(get_key)
      226    0.006    0.000    8.554    0.038 /usr/lib/python3/dist-packages/gpg/gpgme.py:1336(gpgme_get_key)
      226    8.548    0.038    8.548    0.038 {built-in method gpg._gpgme.gpgme_get_key}
     1598    0.022    0.000    6.913    0.004 /usr/lib/python3/dist-packages/gpg/core.py:137(_funcwrap)
      991    0.009    0.000    6.389    0.006 /usr/lib/python3.8/email/__init__.py:40(message_from_bytes)
      991    0.002    0.000    6.377    0.006 /usr/lib/python3.8/email/parser.py:114(parsebytes)
      991    0.096    0.000    6.325    0.006 /usr/lib/python3.8/email/parser.py:59(parsestr)
      991    0.013    0.000    6.229    0.006 /usr/lib/python3.8/email/parser.py:41(parse)
     8301    0.010    0.000    5.830    0.001 /usr/lib/python3.8/email/feedparser.py:173(feed)
      230    0.009    0.000    5.739    0.025 /home/kevin/vcs/once/alot/alot/crypto.py:190(verify_detached)
     9292    0.010    0.000    5.550    0.001 /usr/lib/python3.8/email/feedparser.py:178(_call_parse)
21914/9292    0.444    0.000    5.540    0.001 /usr/lib/python3.8/email/feedparser.py:218(_parsegen)
    20280    0.066    0.000    4.765    0.000 /usr/lib/python3.8/email/message.py:462(get)
        1    0.000    0.000    4.704    4.704 /home/kevin/vcs/once/alot/alot/commands/globals.py:110(apply)
        1    0.000    0.000    4.703    4.703 /home/kevin/vcs/once/alot/alot/ui.py:403(buffer_open)
        1    0.000    0.000    4.703    4.703 /home/kevin/vcs/once/alot/alot/ui.py:460(buffer_focus)
    18233    0.043    0.000    4.675    0.000 /usr/lib/python3.8/email/policy.py:150(header_fetch_parse)
    18693    0.036    0.000    4.653    0.000 /usr/lib/python3.8/email/headerregistry.py:592(__call__)
      230    0.011    0.000    4.344    0.019 /usr/lib/python3/dist-packages/gpg/core.py:501(verify)
    15865    0.038    0.000    4.243    0.000 /usr/lib/python3.8/email/message.py:564(get_content_type)
      230    0.008    0.000    4.242    0.018 /usr/lib/python3/dist-packages/gpg/gpgme.py:1794(gpgme_op_verify)
      230    4.234    0.018    4.234    0.018 {built-in method gpg._gpgme.gpgme_op_verify}
    18693    0.083    0.000    4.153    0.000 /usr/lib/python3.8/email/headerregistry.py:195(__new__)
    17812    0.105    0.000    3.820    0.000 /usr/lib/python3.8/email/headerregistry.py:444(parse)
      456    0.011    0.000    2.758    0.006 /usr/lib/python3/dist-packages/gpg/core.py:185(__init__)
     6384    0.019    0.000    2.730    0.000 /usr/lib/python3/dist-packages/gpg/core.py:164(__setattr__)
      456    0.005    0.000    2.670    0.006 /usr/lib/python3/dist-packages/gpg/core.py:1131(home_dir)
      456    0.006    0.000    2.664    0.006 /usr/lib/python3/dist-packages/gpg/core.py:1349(set_engine_info)
      456    0.015    0.000    2.646    0.006 /usr/lib/python3/dist-packages/gpg/gpgme.py:1069(gpgme_ctx_set_engine_info)
      456    2.631    0.006    2.631    0.006 {built-in method gpg._gpgme.gpgme_ctx_set_engine_info}
        1    0.000    0.000    2.525    2.525 /home/kevin/.config/alot/hooks.py:56(exit)
        1    0.000    0.000    2.525    2.525 /usr/lib/python3.8/subprocess.py:332(call)
        6    0.000    0.000    2.520    0.420 /usr/lib/python3.8/subprocess.py:1074(wait)
        6    0.000    0.000    2.520    0.420 /usr/lib/python3.8/subprocess.py:1772(_wait)
        4    0.000    0.000    2.520    0.630 /usr/lib/python3.8/subprocess.py:1759(_try_wait)
        4    2.520    0.630    2.520    0.630 {built-in method posix.waitpid}
     8195    0.037    0.000    2.106    0.000 /usr/lib/python3.8/email/message.py:588(get_content_maintype)
    17572    0.154    0.000    1.832    0.000 /usr/lib/python3.8/email/_header_value_parser.py:2612(parse_content_type_header)
  1008842    0.274    0.000    1.204    0.000 /usr/lib/python3.8/email/feedparser.py:128(__next__)
    17427    0.044    0.000    1.154    0.000 /usr/lib/python3.8/email/_header_value_parser.py:2544(parse_mime_parameters)
    27655    0.104    0.000    1.088    0.000 /usr/lib/python3.8/email/_header_value_parser.py:2402(get_parameter)
paretje commented 4 years ago

A bisect resulted in 036f30ed7c993a461f1131a358e428461a8d4396

paretje commented 4 years ago

I guess this is the issue:

diff --git a/alot/db/message.py b/alot/db/message.py
index 02c059f6..9f5fafb5 100644
--- a/alot/db/message.py
+++ b/alot/db/message.py
@@ -10,7 +10,7 @@ from datetime import datetime
 from notmuch import NullPointerError

 from . import utils
-from .utils import extract_body
+from .utils import get_body_part, extract_body_part
 from .utils import decode_header
 from .attachment import Attachment
 from .. import helper
@@ -68,6 +68,8 @@ class Message:
         else:
             self._from = '"Unknown" <>'

+        self.mime_part = get_body_part(self.get_email())
+
     def __str__(self):
         """prettyprint the message"""
         aname, aaddress = self.get_author()
@@ -263,8 +265,7 @@ class Message:

     def get_body_text(self):
         """ returns bodystring extracted from this mail """
-        # TODO: allow toggle commands to decide which part is considered body
-        return extract_body(self.get_email())
+        return extract_body_part(self.mime_part)