opening mail "crashes" alot

josch commented 10 years ago

Hi,

I just received the following spam (encoded in base64 to make sure it reaches you byte by byte the same):

UmV0dXJuLVBhdGg6IHJqbkBjYXBzLmFtClJlY2VpdmVkOiAgZnJvbSBzY2kuYW0gKFs1LjE5OS4x
OTcuMTU2XSkgYnkgbXgtaGEud2ViLmRlIChteHdlYjEwOSkgd2l0aCBFU01UUAogKE5lbWVzaXMp
IGlkIDBMZXg5di0xVnpCNU8zb3J2LTAwcW45eSBmb3IgPGouc2NoYXVlckB3ZWIuZGU+OyBUdWUs
IDA1IE5vdgogMjAxMyAwOTowNTo1NCArMDEwMApSZWNlaXZlZDogYnkgbG9jYWxob3N0OyBUdWUs
IDA1IE5vdiAyMDEzIDA5OjAyOjIyICswMTAwCnVqcSBobGQKRnJvbTogIkFubmlrYSBLb2NoIiA8
cmpuQGNhcHMuYW0+ClJlcGx5LVRvOiAiQW5uaWthIEtvY2giIDxyam5AY2Fwcy5hbT4KVG86IG9r
aWtpQHdlYi5kZQpTdWJqZWN0OiBLcmVkaXQgYmlzIDEwMCAwMDAgRXVybyBhdWNoIG9obmUgU2No
dWZhCkRhdGU6IFR1ZSwgMDUgTm92IDIwMTMgMTE6MDA6MjIgKzAzMDAKQ29udGVudC1UcmFuc2Zl
ci1FbmNvZGluZzogN0JpdApDb250ZW50LVR5cGU6IHRleHQvcGxhaW47ClgtQW50aXZpcnVzOiBh
dmFzdCEgKFZQUyAxMzEwMjAtMCwgMjAuMTAuMjAxMyksIE91dGJvdW5kIG1lc3NhZ2UKWC1BbnRp
dmlydXMtU3RhdHVzOiBDbGVhbgpFbnZlbG9wZS1UbzogPGouc2NoYXVlckB3ZWIuZGU+ClgtVUkt
RmlsdGVycmVzdWx0czogdW5rbm93bjo1O1YwMTpLMDpyRWYzZUJaTmFYaz06bThwWFRtMmZlRS9p
V0pINEZleERpYXJKWXkKIGMxeEZWVU54T01McFBzNmV6ZUdNb1JRQ1NyZ0FWL2l4MDdQbUp6MEgx
NE9UUUloMHdkRFJzRzhmaXdCRmc2Tml6Vm9mL1M4WE8KIFBHdEVKWVJ0RjBnYUJBcW0wSWFIeFcr
aERPYk9oYXZ3dWxIN3A0YWhpa1E0aVB4Y3p2TFJxN0hWa2hQWjQ1Qk5aeDdCKzFaU1YKIEw3Nkoy
MUdzVFR3dkoxcU5MMmpRTjJUMGdsV3MxWmsrWjZDMWNvT2VPZ1FNV2ZZdWl0U3poMk1xSE9Cc0VW
S1ZqSXhkQmJ6MnkKIE5rWkZTVTdTSWZKd09GeWFCKythbVMvVzdMaUZ1dDZvQ3VWT2FFdlppeS9F
aytSWUlVbzl1SmhTaEtVRjdDcVJ1elpiUlY2UVAKIEdKNEVTVXVwdGJxRUt0WUNYTzdEOVdYM2Rk
eUtHVDJueDBWb2ViakhkNGVQQ3FWYmQvbXlzdjlIOUh2c1BSOGZpVk53dTZ0MVAKIEpRTHFLNnFD
NUNjK3YzajQ1U3U0MU9RaVBvTVMrLzdTRlQvVjR4ZGRKK0hNRVByTTNJZTJEUW1VaHVFdmZ4bDRF
QlhTY3M0RjQKIDBKVXZsOFU0OTFMaG1sTXcrS05kU2JsUGhHOEwxYnlvOVhHNWZNRHFZRHdETTRI
OEk4Y3RPLzZrUEVzTkN3SVZ3WUVHMC9qR0sKICtjUHhKMXhUUEZMVmoxTFVacklkWGlCNEZFZnNK
dlZKWEZHNnlLMThXUlFvV0RRaHJVVGU5TCs5TkpPSWluOXFBUko3cU1ES2MKIEdZbTJQdGNXR2hJ
MytrbFM5ZFhrTkk4RHp2YllOK2hva1ZRMHR0aXdxT0pPd1B5bDV2ZEdLOUdrMnJXZ2ZhSHpVaC9n
Rmg3Tm8KIHlrdWNYaEtSNmRwY3VYNlJFVTZHNDVFbkdSckZ0azVRTlFOOXh2V0xmSzR1Nk8rcGc5
T1ZBdk9BbE1aaTJiblhKOTc5UFYwWEUKIHNUWHZCZE5qMG5SNjZLTm1SV01sSis0djRkKzdnN3E2
bUZsbXBHTTMyUWVOU0NYUmJQSktqd1JJdmV3VnRPYTVlT2k2azJxcmgKIHdVbTdzNVhkZUN6aTFL
NTBtZ1phdXp5QVM0bEZhMmJTN2hJTHRTdFRLaHo1allDUGlCMWJmTEhrbFg1aG5MWXU4aHQxa0Mx
Q0sKIFZBQmZSOWR4VGRHTEJsajNYYThWclFINTJPdno2TnZvVlNoRWFyQkwvbGExTmFNTUUwOFJi
WGQ0TllQZWdpRk5zQThHQ3g0YlkKIFd5NUQxNHBZSGZZMHhnY0M1Y0k5dHBoeDUwb0ZtYmVmSG8w
Vk5udkI5NnR3YkhFSlhhT0Rhc1VTOXZidXpOSGV3UlM0Z3FrU0IKIENvd21pN1A4Mi9MYVpKeUJT
T1hNVkx4amwwN242VjlKcmFkZlQvUnB0bDBLQjVxRkZJWGxBN2xEaHVnUVV6cnM0aXoxcE9hYjgK
IFFMMUxJL1o2L3FYdlJsdjYvTjM1TkJnUkdTSHNCRU92NDduU3U4WEtiUTA4UDdRS2swNVZ6djZX
OXpyQVlDb09Ha0Z6ZDhCUHgKIDNubXJkUjI5NHNqdTdCVCtMT1pJelpTNUJvNklIV0tFQ29Ca3du
RlVtVnkxS1c5QnNVR1d5UTgvVGdyVUYrWWhSZHdBbnV1YkwKIEcxT3l5RGM1b1ZCT1hZbnhFSlVx
MlEzdFgzNVZBL0JCWmMvNGJOdFpxaHNabThKbmh3Ykw4T1V3M2JzOFhHdlBHWmFyRVNXOUgKIERJ
NEs5dVdmai9lNXM1endhNEIvTWE3NytSYTVUNWZkNUpkZk9uNVV3SnZud2pmVXNkQ2NxWCt6bTVh
Ly8zWWVsakVnUmh0c2kKIE8wQ1BvZUFGa1g2Sm9wSkh5NnFsSVA0MjBkaDFRZktXT2VnVW9hQ2w5
TnRxc0VoNUdiSmFqZmUrSC9jcG13bVl2T3NicmQ0bHEKIHFLSzFObXRMemV3dTYvUHFqeWVNd3ZZ
dXltMjhOclUrYms4K2RaZ0tScEYyZjZGeCtwbFlLbmR0QXdqOU5XWlFIUFlTVGhxRnUKIFpLdGZI
SEN6MlNSR0JSVjJRZEZRaUpmUWc4aS9Cd2hZYy82Zlk0cStkd3JIenJqbW1oWmxNNHB4NERUR0tj
Tm5PZ1BKVGh2SlMKIHE1R2pTUUtEbXBvU1kwS29mWHhSNFR3QVd0WEthbFI1CgpHdXRlbiBUYWcs
CgpTb2ZvcnRLcmVkaXQgdm9uIDIwMDAgYmlzIHp1IDEwMCAwMDAgRXVybwoKQXVjaCBvaG5lIFNj
aHVmYSBt9mdsaWNoIQoKQmVzdWNoZW4gU2llIHVuc2VyZSBXZWJzZWl0ZSB1bnRlcjoKCnd3dy5h
a3JlZGl0OS5jb20KCgoKCgoKTmV3cyBhYm1lbGRlbgoKd3d3LmFrcmVkaXQ5LmNvbS9hc20K

The decoded data is has an md5sum of 8248c9c326482cd808ab77b336f6100d

When I try to open the thread containing this message in alot (I'm running the master branch) then I get the following error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "alot/init.py", line 187, in main
    UI(dbman, cmd)
  File "alot/ui.py", line 85, in __init__
    self.mainloop.run()
  File "/usr/lib/python2.7/dist-packages/urwid/main_loop.py", line 272, in run
    self.screen.run_wrapper(self._run)
  File "/usr/lib/python2.7/dist-packages/urwid/raw_display.py", line 242, in run_wrapper
    return fn()
  File "/usr/lib/python2.7/dist-packages/urwid/main_loop.py", line 337, in _run 
    self.event_loop.run()
  File "/usr/lib/python2.7/dist-packages/urwid/main_loop.py", line 1256, in wrapper
    rval = f(*args,**kargs)
  File "/usr/lib/python2.7/dist-packages/urwid/main_loop.py", line 1184, in _twisted_idle_callback
    callback()
  File "/usr/lib/python2.7/dist-packages/urwid/main_loop.py", line 549, in entering_idle
    self.draw_screen()
  File "/usr/lib/python2.7/dist-packages/urwid/main_loop.py", line 563, in draw_screen
    canvas = self._topmost_widget.render(self.screen_size, focus=True)
  File "/usr/lib/python2.7/dist-packages/urwid/widget.py", line 141, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python2.7/dist-packages/urwid/decoration.py", line 225, in render
    canv = self._original_widget.render(size, focus=focus)
  File "/usr/lib/python2.7/dist-packages/urwid/widget.py", line 141, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python2.7/dist-packages/urwid/container.py", line 1058, in render
    focus and self.focus_part == 'body')
  File "alot/buffers.py", line 407, in render
    return self.body.render(size, focus)
  File "/usr/lib/python2.7/dist-packages/urwid/widget.py", line 141, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python2.7/dist-packages/urwid/widget.py", line 1751, in render 
    canv = get_delegate(self).render(size, focus=focus)
  File "/usr/lib/python2.7/dist-packages/urwid/widget.py", line 141, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python2.7/dist-packages/urwid/listbox.py", line 457, in render 
    (maxcol, maxrow), focus=focus)
  File "/usr/lib/python2.7/dist-packages/urwid/listbox.py", line 402, in calculate_visible
    next, pos = self.body.get_next( pos )
  File "alot/foreign/urwidtrees/widgets.py", line 80, in get_next
    return self._get(self._tree.next_position(pos))
  File "alot/foreign/urwidtrees/widgets.py", line 67, in _get
    res = self[pos], pos
  File "alot/foreign/urwidtrees/lru_cache.py", line 102, in wrapper
    result = user_function(*args, **kwds)
  File "alot/foreign/urwidtrees/widgets.py", line 53, in __getitem__
    entry = self._tree.get_decorated(pos)
  File "alot/foreign/urwidtrees/nested.py", line 86, in get_decorated
    return self._get_decorated_entry(self._tree, pos)
  File "alot/foreign/urwidtrees/nested.py", line 82, in _get_decorated_entry
    entry = tree.decorate(pos[0], entry, is_first=isf)
  File "alot/foreign/urwidtrees/decoration.py", line 475, in decorate
    line = urwid.Columns(cols, box_columns=range(len(cols))[:-1])
  File "/usr/lib/python2.7/dist-packages/urwid/container.py", line 1736, in __init__
    if focus_column is None and w.selectable():
AttributeError: 'NoneType' object has no attribute 'selectable'

pazz commented 10 years ago

Can you confirm that you meet the requirements regarding the urwid version?

josch commented 10 years ago

Hi, my alot setup.py says urwid (>=1.1.0) is required. My python-urwid version is:

$ dpkg -l | grep urwid
ii  python-urwid                          1.1.1-1                              amd64        curses-based UI/widget library for Python

pazz commented 10 years ago

maybe you have an older version installed locally? what does

python -c "import urwid; print urwid.__version__"

say?

josch commented 10 years ago

alot is one of the only two pieces of software I'm using which I have not installed through my package manager so this would be unlikely. Anyways, here the output:

$ python -c "import urwid; print urwid.__version__"
1.1.1

pazz commented 10 years ago

Thanks. Do you see anything out of the ordinary in this mail? (how do i quickly import this base64 encoded blob for debugging?)

josch commented 10 years ago

Not really.

Here is the email in plaintext copypasted:

Return-Path: rjn@caps.am
Received:  from sci.am ([5.199.197.156]) by mx-ha.web.de (mxweb109) with ESMTP
 (Nemesis) id 0Lex9v-1VzB5O3orv-00qn9y for <j.schauer@web.de>; Tue, 05 Nov
 2013 09:05:54 +0100
Received: by localhost; Tue, 05 Nov 2013 09:02:22 +0100
ujq hld
From: "Annika Koch" <rjn@caps.am>
Reply-To: "Annika Koch" <rjn@caps.am>
To: okiki@web.de
Subject: Kredit bis 100 000 Euro auch ohne Schufa
Date: Tue, 05 Nov 2013 11:00:22 +0300
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain;
X-Antivirus: avast! (VPS 131020-0, 20.10.2013), Outbound message
X-Antivirus-Status: Clean
Envelope-To: <j.schauer@web.de>
X-UI-Filterresults: unknown:5;V01:K0:rEf3eBZNaXk=:m8pXTm2feE/iWJH4FexDiarJYy
 c1xFVUNxOMLpPs6ezeGMoRQCSrgAV/ix07PmJz0H14OTQIh0wdDRsG8fiwBFg6NizVof/S8XO
 PGtEJYRtF0gaBAqm0IaHxW+hDObOhavwulH7p4ahikQ4iPxczvLRq7HVkhPZ45BNZx7B+1ZSV
 L76J21GsTTwvJ1qNL2jQN2T0glWs1Zk+Z6C1coOeOgQMWfYuitSzh2MqHOBsEVKVjIxdBbz2y
 NkZFSU7SIfJwOFyaB++amS/W7LiFut6oCuVOaEvZiy/Ek+RYIUo9uJhShKUF7CqRuzZbRV6QP
 GJ4ESUuptbqEKtYCXO7D9WX3ddyKGT2nx0VoebjHd4ePCqVbd/mysv9H9HvsPR8fiVNwu6t1P
 JQLqK6qC5Cc+v3j45Su41OQiPoMS+/7SFT/V4xddJ+HMEPrM3Ie2DQmUhuEvfxl4EBXScs4F4
 0JUvl8U491LhmlMw+KNdSblPhG8L1byo9XG5fMDqYDwDM4H8I8ctO/6kPEsNCwIVwYEG0/jGK
 +cPxJ1xTPFLVj1LUZrIdXiB4FEfsJvVJXFG6yK18WRQoWDQhrUTe9L+9NJOIin9qARJ7qMDKc
 GYm2PtcWGhI3+klS9dXkNI8DzvbYN+hokVQ0ttiwqOJOwPyl5vdGK9Gk2rWgfaHzUh/gFh7No
 ykucXhKR6dpcuX6REU6G45EnGRrFtk5QNQN9xvWLfK4u6O+pg9OVAvOAlMZi2bnXJ979PV0XE
 sTXvBdNj0nR66KNmRWMlJ+4v4d+7g7q6mFlmpGM32QeNSCXRbPJKjwRIvewVtOa5eOi6k2qrh
 wUm7s5XdeCzi1K50mgZauzyAS4lFa2bS7hILtStTKhz5jYCPiB1bfLHklX5hnLYu8ht1kC1CK
 VABfR9dxTdGLBlj3Xa8VrQH52Ovz6NvoVShEarBL/la1NaMME08RbXd4NYPegiFNsA8GCx4bY
 Wy5D14pYHfY0xgcC5cI9tphx50oFmbefHo0VNnvB96twbHEJXaODasUS9vbuzNHewRS4gqkSB
 Cowmi7P82/LaZJyBSOXMVLxjl07n6V9JradfT/Rptl0KB5qFFIXlA7lDhugQUzrs4iz1pOab8
 QL1LI/Z6/qXvRlv6/N35NBgRGSHsBEOv47nSu8XKbQ08P7QKk05Vzv6W9zrAYCoOGkFzd8BPx
 3nmrdR294sju7BT+LOZIzZS5Bo6IHWKECoBkwnFUmVy1KW9BsUGWyQ8/TgrUF+YhRdwAnuubL
 G1OyyDc5oVBOXYnxEJUq2Q3tX35VA/BBZc/4bNtZqhsZm8JnhwbL8OUw3bs8XGvPGZarESW9H
 DI4K9uWfj/e5s5zwa4B/Ma77+Ra5T5fd5JdfOn5UwJvnwjfUsdCcqX+zm5a//3YeljEgRhtsi
 O0CPoeAFkX6JopJHy6qlIP420dh1QfKWOegUoaCl9NtqsEh5GbJajfe+H/cpmwmYvOsbrd4lq
 qKK1NmtLzewu6/PqjyeMwvYuym28NrU+bk8+dZgKRpF2f6Fx+plYKndtAwj9NWZQHPYSThqFu
 ZKtfHHCz2SRGBRV2QdFQiJfQg8i/BwhYc/6fY4q+dwrHzrjmmhZlM4px4DTGKcNnOgPJThvJS
 q5GjSQKDmpoSY0KofXxR4TwAWtXKalR5

Guten Tag,

SofortKredit von 2000 bis zu 100 000 Euro

Auch ohne Schufa möglich!

Besuchen Sie unsere Webseite unter:

www.akredit9.com

News abmelden

www.akredit9.com/asm

I encode/decode base64 using the base64 program which is part of the coreutils package in Debian/Ubuntu/Mint.

Otherwise any web based encode/decoder will do the same job. I just wanted to avoid any mangling with the text by github.

sjackso commented 10 years ago

I see a line of unindented cruft within the headers:

...
Received: by localhost; Tue, 05 Nov 2013 09:02:22 +0100
ujq hld
From: "Annika Koch" <rjn@caps.am>
...

I have done no tests myself, but perhaps this is a source of trouble?

pazz commented 10 years ago

yes, this is certainly a malformed mail because of this string ujq hld in line 6. I imported the base64 encoded mail to my local notmuch index and for me, alot does not die when i open this thread. In thread mode, it shows the one message, with Annika Koch (08:00am) as summary and considers the whole mail as body text from line 6 onwards.

My guess is that we have different versions of notmuch running, mine (master, HEAD: 48f2b3f1bc279f61b78d2cb4b19e8b05e607b125) being less strict than yours. It seems that in my case, libnotmuch extracts the value for the From header from this mail, although the corresponding line is only after the gibberish in line 6. I suspect that in your case, notmuch did not extract this value, resulting in a message for which the indexed From-value is None. I would not be surprised if such a scenario causes trouble for alot.

Could you upgrade notmuch and bindings to current upstream versions and re-index? Does this fix the issue? Otherwise, we'll need to cook up a code snipped to verify my guess :)

pazz / alot

opening mail "crashes" alot #667