search

pazz / alot

Terminal-based Mail User Agent
GNU General Public License v3.0
696 stars 164 forks source link

UnicodeEncodeError: 'latin-1' codec can't encode character '\uf0e0' in position 0: ordinal not in range(256) #1557

Open teto opened 3 years ago

teto commented 3 years ago

Before you submit a bug report, please make sure that the issue still exists on the master branch!

Describe the bug On start I get


Traceback (most recent call last):
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/bin/.alot-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/__main__.py", line 137, in main
    UI(dbman, cmdstring)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 146, in __init__
    self.mainloop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 287, in run
    self._run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 385, in _run
    self.event_loop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1346, in run
    reraise(*exc_info)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/compat.py", line 58, in reraise
    raise value
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1360, in wrapper
    rval = f(*args,**kargs)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1319, in _twisted_idle_callback
    callback()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 574, in entering_idle
    self.draw_screen()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 589, in draw_screen
    self.screen.draw_screen(self.screen_size, canvas)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 868, in draw_screen
    self.write(l)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 286, in write
    self._term_output_file.write(data)
UnicodeEncodeError: 'latin-1' codec can't encode character '\uf0e0' in position 0: ordinal not in range(256)

Software Versions

To Reproduce Steps to reproduce the behaviour:

  1. for me just type "alot"

Error Log Please include all error information from the log file. relevant part from the log

Traceback (most recent call last):
  File "/nix/store/i9lzacdfrwlj1ayw551c016s0fq71p7j-python3-3.8.6/lib/python3.8/logging/__init__.py", line 1084, in emit
    stream.write(msg + self.terminator)
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 30-34: ordinal not in range(256)
Call stack:
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/bin/.alot-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/__main__.py", line 137, in main
    UI(dbman, cmdstring)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 146, in __init__
    self.mainloop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 287, in run
    self._run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 385, in _run
    self.event_loop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1341, in run
    self.reactor.run()
  File "/nix/store/012k72gbxdbd76j0cckpz3sn51bsc4mx-python3.8-Twisted-20.3.0/lib/python3.8/site-packages/twisted/internet/asyncioreactor.py", line 267, in run
    self._asyncioEventloop.run_forever()
  File "/nix/store/i9lzacdfrwlj1ayw551c016s0fq71p7j-python3-3.8.6/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/nix/store/i9lzacdfrwlj1ayw551c016s0fq71p7j-python3-3.8.6/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/nix/store/i9lzacdfrwlj1ayw551c016s0fq71p7j-python3-3.8.6/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 277, in apply_commandline
    await apply_this_command(c)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 723, in apply_command
    cmd.apply(self)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/commands/globals.py", line 125, in apply
    ui.buffer_open(buffers.SearchBuffer(ui, self.query,
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 417, in buffer_open
    self.buffer_focus(buf)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 477, in buffer_focus
    self.update()
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 668, in update
    self.mainloop.draw_screen()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 588, in draw_screen
    canvas = self._topmost_widget.render(self.screen_size, focus=True)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/widget.py", line 145, in cached_render
    canv = fn(self, size, focus=focus)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/decoration.py", line 226, in render
    canv = self._original_widget.render(size, focus=focus)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/widget.py", line 145, in cached_render
    canv = fn(self, size, focus=focus)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/container.py", line 1089, in render
    body = self.body.render((maxcol, maxrow-ftrim-htrim),
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/buffers/buffer.py", line 19, in render
    return self.body.render(size, focus)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/widget.py", line 145, in cached_render
    canv = fn(self, size, focus=focus)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/listbox.py", line 439, in render
    middle, top, bottom = self.calculate_visible(
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/listbox.py", line 322, in calculate_visible
    self._set_focus_complete( (maxcol, maxrow), focus )
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/listbox.py", line 686, in _set_focus_complete
    return self._set_focus_first_selectable(
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/listbox.py", line 656, in _set_focus_first_selectable
    middle, top, bottom = self.calculate_visible(
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/listbox.py", line 385, in calculate_visible
    next, pos = self._body.get_next( pos )
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/walker.py", line 46, in get_next
    return self._get_at_pos(start_from + self.direction)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/walker.py", line 72, in _get_at_pos
    widget = self._get_next_item()
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/walker.py", line 85, in _get_next_item
    next_widget = self.containerclass(next_obj, **self.kwargs)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/widgets/search.py", line 26, in __init__
    self.rebuild()
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/widgets/search.py", line 60, in rebuild
    width, part = build_text_part(partname, self.thread,
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/widgets/search.py", line 145, in build_text_part
    content = prepare_string(name, thread, maxw)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/widgets/search.py", line 212, in prepare_string
    s = content(thread)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/widgets/search.py", line 177, in prepare_authors_string
    return thread.get_authors_string() or '(None)'
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 199, in get_authors_string
    for aname, aaddress in self.get_authors():
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 162, in get_authors
    msgs = sorted(self.get_messages().keys(),
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 248, in get_messages
    self._toplevel_messages.append(accumulate(self._messages,
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 243, in accumulate
    acc[M].append(accumulate(acc, m))
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 243, in accumulate
    acc[M].append(accumulate(acc, m))
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 243, in accumulate
    acc[M].append(accumulate(acc, m))
  [Previous line repeated 2 more times]
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 240, in accumulate
    M = Message(self._dbman, msg, thread=self)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/message.py", line 59, in __init__
    sender = decode_header(msg.header('From'))
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/utils.py", line 539, in decode_header
    logging.debug("unquoted header: |%s|", header)
Message: 'unquoted header: |%s|'
Arguments: ('\u0410\u0440\u0442\u0451\u043c \u041c\u0443\u0445\u0430\u043c\u0435\u0434-\u041a\u0430\u0440\u0438\u043c\u043e\u0432 \u041c\u041f\u0411-802 <XXXXX@gmail.com>',)
Traceback (most recent call last):
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/bin/.alot-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/__main__.py", line 137, in main
    UI(dbman, cmdstring)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 146, in __init__
    self.mainloop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 287, in run
    self._run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 385, in _run
    self.event_loop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1346, in run
    reraise(*exc_info)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/compat.py", line 58, in reraise
    raise value
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1360, in wrapper
    rval = f(*args,**kargs)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1319, in _twisted_idle_callback
    callback()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 574, in entering_idle
    self.draw_screen()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 589, in draw_screen
    self.screen.draw_screen(self.screen_size, canvas)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 868, in draw_screen
    self.write(l)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 286, in write
    self._term_output_file.write(data)
UnicodeEncodeError: 'latin-1' codec can't encode character '\uf0e0' in position 0: ordinal not in range(256)
lucc commented 3 years ago

This looks like it is related to some message in your initial buffer. Can you try to find the message and confirm by starting alot with alot search id:my-special-message. Assuming you did not change bug_on_exit or initial_command you might use notmuch search --output messages tag:inbox AND NOT tag:killed | xargs -t -I{} -n1 alot search {} ';' exit in your shell to figure out which message is at fault.

teto commented 3 years ago

your command gives me

notmuch search --output messages tag:inbox AND NOT tag:killed | xargs -t -I{} -n1 alot search {} ';' exit                                                                                 ~
alot search id:99aa6841f5fd378b65518716f23457f6b52aea0c.1604845519.git.geliangtang@gmail.com ';' exit 
Traceback (most recent call last):
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/bin/.alot-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/__main__.py", line 137, in main
    UI(dbman, cmdstring)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 146, in __init__
    self.mainloop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 287, in run
    self._run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 385, in _run
    self.event_loop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1346, in run
    reraise(*exc_info)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/compat.py", line 58, in reraise
    raise value
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1360, in wrapper
    rval = f(*args,**kargs)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 417, in <lambda>
    event_loop, callback, self.get_available_raw_input())
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 449, in get_available_raw_input
    codes = self._get_gpm_codes() + self._get_keyboard_codes()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 523, in _get_keyboard_codes
    code = self._getch_nodelay()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 661, in _getch_nodelay
    return self._getch(0)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 571, in _getch
    return ord(os.read(fd, 1))
TypeError: ord() expected a character, but string of length 0 found

And I've uploaded the content of notmuch show id:99aa6841f5fd378b65518716f23457f6b52aea0c.1604845519.git.geliangtang@gmail.com > msg.txt here: msg.txt

lucc commented 3 years ago

That is the file as rendered by notmuch(1). Please upload the raw file from your mail folder. You can find it with notmuch search --output files id:99aa6841f5fd378b65518716f23457f6b52aea0c.1604845519.git.geliangtang@gmail.com.

teto commented 3 years ago

sry, here you go: msg.txt note that the message appears 3 times in the output

notmuch search --output files id:99aa6841f5fd378b65518716f23457f6b52aea0c.1604845519.git.geliangtang@gmail.com                                                                           
/home/teto/maildir/gmail/[Gmail]/Tous les messages/cur/1604851446.4383_1515.jedha,U=22591:2,S
/home/teto/maildir/gmail/Inbox/cur/1604851319.4383_514.jedha,U=106227:2,S
/home/teto/maildir/gmail/Inbox/cur/1605397110.26667_3953.jedha,U=173701:2,S
lucc commented 3 years ago

That message looks harmless (it claims to be ascii and actually is ascii). Can you confirm that alot search id:99aa6841f5fd378b65518716f23457f6b52aea0c.1604845519.git.geliangtang@gmail.com produces the original error?

I just reread your https://github.com/pazz/alot/issues/1557#issuecomment-744059973 and tested it. That is actually another error and not your original one. That was my bad I am sorry, alot seems to not like it when stdin is not a terminal. So we have to search the offending message like this:

for q in $(notmuch search --output messages tag:inbox AND NOT tag:killed); do
  echo $q
  alot search $q \; exit
done

If you find a message please also confirm that it reproduces your original error.

lucc commented 3 years ago

I also reread the first exception and it seems to fail when encoding to latin1 (instead of decoding from latin1). So what is your $TERM and $LANG just in case.