UnicodeEncodeError

[Fjodor42]

I seem to be getting a lot of the following in the log on my instance - what can I do about it?

Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: 2017-04-30 12:48:55 [leap.bitmask.mail.incoming.service] ERROR [Failure instance: Traceback: <type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode characters in position 31-32: ordinal not in range(128)
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:317:addCallback
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:306:addCallbacks
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:588:_runCallbacks
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1274:unwindGenerator
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: --- <exception caught here> ---
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1128:_inlineCallbacks
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/share/python/pixelated-user-agent/src/leap.bitmask/src/leap/bitmask/mail/incoming/service.py:673:_maybe_extract_keys
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/message.py:137:as_string
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/generator.py:83:flatten
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/generator.py:108:_write
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/generator.py:134:_dispatch
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/generator.py:203:_handle_multipart
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/generator.py:83:flatten
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/generator.py:108:_write
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/generator.py:134:_dispatch
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/generator.py:288:_handle_message
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/generator.py:83:flatten
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/generator.py:115:_write
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/generator.py:164:_write_headers
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/header.py:183:__init__
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: /usr/lib/python2.7/email/header.py:272:append
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: ]
Apr 30 12:48:55 mail2 pixelated-user-agent[27199]: 2017-04-30 12:48:55 [leap.bitmask.mail.incoming.service] INFO there are 2 mails

[tuliocasagrande]

Hello @Fjodor42, thank you for opening this issue. Are you able to reproduce the error or you're getting it randomly?

[Fjodor42]

Hello, and thank you for looking into this!

To the best of my knowledge, it happens at some points in time when certain emails, that I'm not sure of which are, are coming in.

It would also seem likely that this is related to another error that I get quite frequently, of which the following is an example:

May  2 22:34:27 mail2 pixelated-user-agent[14086]: 2017-05-02 22:34:27 [pixelated.adapter.mailstore.body_parser] WARN Failed to encode content for charset us-a
scii. Ignoring invalid chars: 'ascii' codec can't encode character u'\ufffd' in position 731: ordinal not in range(128)
May  2 22:34:27 mail2 pixelated-user-agent[14086]: 2017-05-02 22:34:27 [pixelated.adapter.mailstore.body_parser] WARN Failed to encode content for charset us-a
scii. Ignoring invalid chars: 'ascii' codec can't encode character u'\ufffd' in position 732: ordinal not in range(128)
May  2 22:34:27 mail2 pixelated-user-agent[14086]: 2017-05-02 22:34:27 [pixelated.adapter.mailstore.body_parser] WARN Failed to encode content for charset us-a
scii. Ignoring invalid chars: 'ascii' codec can't encode characters in position 185-188: ordinal not in range(128)
May  2 22:34:27 mail2 pixelated-user-agent[14086]: 2017-05-02 22:34:27 [pixelated.adapter.mailstore.body_parser] WARN Failed to encode content for charset us-a
scii. Ignoring invalid chars: 'ascii' codec can't encode character u'\ufffd' in position 729: ordinal not in range(128)
May  2 22:34:27 mail2 pixelated-user-agent[14086]: 2017-05-02 22:34:27 [pixelated.adapter.mailstore.body_parser] WARN Failed to encode content for charset us-a
scii. Ignoring invalid chars: 'ascii' codec can't encode character u'\ufffd' in position 1097: ordinal not in range(128)

[Fjodor42]

Probably related, from just now:

May 9 13:50:28 mail2 pixelated-user-agent[16166]: 2017-05-09 13:50:28 [twisted.python.log] INFO "0.0.0.0" - - [09/May/2017:11:50:28 +0000] "GET /mails?q=tag%3A%22inbox%22&p=1&w=25 HTTP/1.1" 200 54481 "https://mail.molgaard.org/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0" May 9 13:50:28 mail2 pixelated-user-agent[16166]: 2017-05-09 13:50:28 [pixelated.adapter.mailstore.body_parser] WARN Failed to encode content for charset us-ascii. Ignoring invalid chars: 'ascii' codec can't encode character u'\ufffd' in position 744: ordinal not in range(128) May 9 13:50:28 mail2 pixelated-user-agent[16166]: 2017-05-09 13:50:28 [pixelated.adapter.mailstore.body_parser] WARN Failed to encode content for charset us-ascii. Ignoring invalid chars: 'ascii' codec can't encode character u'\ufffd' in position 732: ordinal not in range(128) May 9 13:50:28 mail2 pixelated-user-agent[16166]: 2017-05-09 13:50:28 [pixelated.adapter.listeners.mailbox_indexer_listener] ERROR FirstError[#27, [Failure instance: Traceback: <type 'exceptions.LookupError'>: unknown encoding: y May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:434:errback May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:501:_startRunCallbacks May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:588:_runCallbacks May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1184:gotResult May 9 13:50:28 mail2 pixelated-user-agent[16166]: --- <exception caught here> --- May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1126:_inlineCallbacks May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/python/failure.py:389:throwExceptionIntoGenerator May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/pixelated_user_agent-0.1-py2.7.egg/pixelated/adapter/mailstore/leap_mailstore.py:201:get_mail May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1126:_inlineCallbacks May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/python/failure.py:389:throwExceptionIntoGenerator May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/pixelated_user_agent-0.1-py2.7.egg/pixelated/adapter/mailstore/leap_mailstore.py:315:_leap_message_to_leap_mail May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/twisted/internet/defer.py:1128:_inlineCallbacks May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/pixelated_user_agent-0.1-py2.7.egg/pixelated/adapter/mailstore/leap_mailstore.py:336:_raw_message_body May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/pixelated_user_agent-0.1-py2.7.egg/pixelated/adapter/mailstore/body_parser.py:42:parsed_content May 9 13:50:28 mail2 pixelated-user-agent[16166]: /usr/share/python/pixelated-user-agent/local/lib/python2.7/site-packages/pixelated_user_agent-0.1-py2.7.egg/pixelated/adapter/mailstore/body_parser.py:61:_serialize_for_parser May 9 13:50:28 mail2 pixelated-user-agent[16166]: ]] May 9 13:50:29 mail2 pixelated-user-agent[16166]: 2017-05-09 13:50:29 [leap.bitmask.mail.incoming.service] INFO there are 1 mails

[Fjodor42]

Any ideas?