sj26 / mailcatcher

Catches mail and serves it through a dream.
http://mailcatcher.me
MIT License
6.31k stars 578 forks source link

Encoding::UndefinedConversionError - "\xD7" from ASCII-8BIT to UTF-8: #339

Closed 4refael closed 3 years ago

4refael commented 7 years ago
Starting MailCatcher
==> smtp://127.0.0.1:1025
==> http://127.0.0.1:1080
==> SMTP: Received message from '<noreply@example.com>' (909 bytes)
*** Error sending message through websocket: {"id"=>1, "sender"=>"<noreply@examp
le.com>", "recipients"=>["<sdafasdfasd@asdfsdf.com>"], "subject"=>"\xD7\x94\xD7\
xA8\xD7\xA9\xD7 \xD7\x94 \xD7\x91\xD7 \xD7\xAA\xD7\xA8 sugar10.local"
, "source"=>"To: sdafasdfasd@asdfsdf.com\r\nSubject: \xD7\x94\xD7\xA8\xD7\xA9\xD
7 \xD7\x94 \xD7\x91\xD7 \xD7\xAA\xD7\xA8 sugar10.local\r\nX-PHP-Origi
nating-Script: 0:Email.php\r\nUser-Agent: CodeIgniter\r\nDate: Thu, 11 May 2017
13:17:14 +0300\r\nFrom: \"test\" <noreply@example.com>\r\nReturn-Path: <noreply@
example.com>\r\nReply-To: \"noreply@example.com\" <noreply@example.com>\r\nX-Sen
der: noreply@example.com\r\nX-Mailer: xampp\r\nX-Priority: 3 (Normal)\r\nMessage
-ID: <59143a2a8d25f@example.com>\r\nMime-Version: 1.0\r\nContent-Type: text/plai
n; charset=utf-8\r\nContent-Transfer-Encoding: 8bit\r\n\r\n\xD7\xA9\xD7\x9C\xD7\
x95\xD7\x9D sadasdas,\r\n\r\n\xD7\xAA\xD7\x95\xD7\x93\xD7\x94 \xD7\xA9\xD7\xA0\x
D7\xA8\xD7\xA9\xD7\x9E\xD7\xAA \xD7\x91\xD7\x90\xD7\xAA\xD7\xA8 sugar
10.local!\r\n\r\n\r\n\xD7\xA2\xD7\x9C \xD7\x9E\xD7\xA0\xD7\xAA \xD7\x9C\xD7\x94\
xD7\xA9\xD7\x9C\xD7\x99\xD7\x9D \xD7\x90\xD7\xAA \xD7\xAA\xD7\x94\xD7\x9C\xD7\x9
9\xD7\x9A \xD7\x94\xD7\x94\xD7\xA8\xD7\xA9\xD7\x9E\xD7\x94, \xD7\xA2\xD7\x9C\xD7
\x99\xD7\x9A \xD7\x9C\xD7\x90\xD7\x9E\xD7\xAA\r\n\xD7\x90\xD7\xAA \xD7\x94\xD7\x
93\xD7\x95\xD7\x90\"\xD7\x9C \xD7\xA9\xD7\x9C\xD7\x9A \xD7\x91\xD7\x9C\xD7\x97\x
D7\x99\xD7\xA6\xD7\x94 \xD7\xA2\xD7\x9C \xD7\x94\xD7\xA7\xD7\x99\xD7\xA9\xD7\x95
\xD7\xA8 \xD7\x94\xD7\x91\xD7\x90:\r\n\r\nPlease follow the link to confirm your
 registration\r\nhttp://sugar10.local/users/confirm/ca2c2c7034\r\n\r\
n\r\n\xD7\x91\xD7\x91\xD7\xA8\xD7\x9B\xD7\x94,\r\n\xD7\xA6\xD7\x95\xD7\x95\xD7\x
AA \xD7\x90\xD7\xAA\xD7\xA8 \xD7\xA9\xD7\x95\xD7\x92\xD7\xA8 \xD7\x93\xD7\x93\xD
7\x99\r\n\r\n", "size"=>"909", "type"=>"text/plain", "created_at"=>#<DateTime: 2
017-05-11T10:17:17+00:00 ((2457885j,37037s,0n),+0s,2299161j)>}
    Exception: "\xD7" from ASCII-8BIT to UTF-8
    Backtrace:
       C:/Ruby23-x64/lib/ruby/2.3.0/json/common.rb:224:in `encode'
       C:/Ruby23-x64/lib/ruby/2.3.0/json/common.rb:224:in `generate'
       C:/Ruby23-x64/lib/ruby/2.3.0/json/common.rb:224:in `generate'
       mailcatcher (0.6.5) lib/mail_catcher/web/application.rb:69:in `block (3 l
evels) in <class:Application>'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block (3 levels) in push'

       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `each'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block (2 levels) in push'

       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `each'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block in push'
       eventmachine (1.0.9.1) lib/eventmachine.rb:256:in `schedule'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `push'
       mailcatcher (0.6.5) lib/mail_catcher/mail.rb:57:in `block in add_message'

       eventmachine (1.0.9.1) lib/eventmachine.rb:981:in `block in run_deferred_
callbacks'
       eventmachine (1.0.9.1) lib/eventmachine.rb:978:in `times'
       eventmachine (1.0.9.1) lib/eventmachine.rb:978:in `run_deferred_callbacks
'
       eventmachine (1.0.9.1) lib/eventmachine.rb:193:in `run_machine'
       eventmachine (1.0.9.1) lib/eventmachine.rb:193:in `run'
       mailcatcher (0.6.5) lib/mail_catcher.rb:173:in `run!'
       mailcatcher (0.6.5) bin/mailcatcher:5:in `<top (required)>'
       C:/Ruby23-x64/bin/mailcatcher:22:in `load'
       C:/Ruby23-x64/bin/mailcatcher:22:in `<main>'
    Please submit this as an issue at http://github.com/sj26/mailcatcher/issues
2017-05-11 13:17:21 - Encoding::UndefinedConversionError - "\xD7" from ASCII-8BI
T to UTF-8:
        C:/Ruby23-x64/lib/ruby/2.3.0/json/common.rb:224:in `encode'
        C:/Ruby23-x64/lib/ruby/2.3.0/json/common.rb:224:in `generate'
        C:/Ruby23-x64/lib/ruby/2.3.0/json/common.rb:224:in `generate'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/mailcatcher-0.6.5/lib/mail_catche
r/web/application.rb:81:in `block in <class:Application>'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1611:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1611:in `block in compile!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:975:in `block (3 levels) in route!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:994:in `route_eval'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:975:in `block (2 levels) in route!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1015:in `block in process_route'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1013:in `catch'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1013:in `process_route'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:973:in `block in route!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:972:in `each'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:972:in `route!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1085:in `block in dispatch!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1067:in `block in invoke'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1067:in `catch'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1067:in `invoke'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1082:in `dispatch!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:907:in `block in call!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1067:in `block in invoke'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1067:in `catch'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1067:in `invoke'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:907:in `call!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:895:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/pr
otection/xss_header.rb:18:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/pr
otection/path_traversal.rb:16:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/pr
otection/json_csrf.rb:18:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/pr
otection/base.rb:49:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/pr
otection/base.rb:49:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/pr
otection/frame_options.rb:31:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-1.6.6/lib/rack/nulllogger.rb
:9:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-1.6.6/lib/rack/head.rb:13:in
 `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:182:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:2013:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1487:in `block in call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1787:in `synchronize'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb
:1487:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-1.6.6/lib/rack/urlmap.rb:66:
in `block in call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-1.6.6/lib/rack/urlmap.rb:50:
in `each'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-1.6.6/lib/rack/urlmap.rb:50:
in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/rack-1.6.6/lib/rack/builder.rb:15
3:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/mailcatcher-0.6.5/lib/mail_catche
r/web.rb:19:in `call'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/thin-1.5.1/lib/thin/connection.rb
:81:in `block in pre_process'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/thin-1.5.1/lib/thin/connection.rb
:79:in `catch'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/thin-1.5.1/lib/thin/connection.rb
:79:in `pre_process'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/thin-1.5.1/lib/thin/connection.rb
:54:in `process'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/thin-1.5.1/lib/thin/connection.rb
:39:in `receive_data'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/eventmachine-1.0.9.1/lib/eventmac
hine.rb:193:in `run_machine'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/eventmachine-1.0.9.1/lib/eventmac
hine.rb:193:in `run'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/mailcatcher-0.6.5/lib/mail_catche
r.rb:173:in `run!'
        C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/mailcatcher-0.6.5/bin/mailcatcher
:5:in `<top (required)>'
        C:/Ruby23-x64/bin/mailcatcher:22:in `load'
        C:/Ruby23-x64/bin/mailcatcher:22:in `<main>'
jgrau commented 7 years ago

Yup. I am having the same issue.

caugner commented 7 years ago

Similar issue here, running mailcatcher using Ruby 2.3.3 (x64) on Windows 10:

    Exception: "\xC3" from ASCII-8BIT to UTF-8
    Backtrace:
       json (2.1.0) lib/json/common.rb:224:in `encode'
       json (2.1.0) lib/json/common.rb:224:in `generate'
       json (2.1.0) lib/json/common.rb:224:in `generate'
       mailcatcher (0.6.5) lib/mail_catcher/web/application.rb:69:in `block (3 levels) in <class:Application>'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block (3 levels) in push'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `each'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block (2 levels) in push'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `each'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block in push'
       eventmachine (1.0.9.1) lib/eventmachine.rb:256:in `schedule'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `push'
       mailcatcher (0.6.5) lib/mail_catcher/mail.rb:57:in `block in add_message'
       eventmachine (1.0.9.1) lib/eventmachine.rb:981:in `block in run_deferred_callbacks'
       eventmachine (1.0.9.1) lib/eventmachine.rb:978:in `times'
       eventmachine (1.0.9.1) lib/eventmachine.rb:978:in `run_deferred_callbacks'
       eventmachine (1.0.9.1) lib/eventmachine.rb:193:in `run_machine'
       eventmachine (1.0.9.1) lib/eventmachine.rb:193:in `run'
       mailcatcher (0.6.5) lib/mail_catcher.rb:173:in `run!'
       mailcatcher (0.6.5) bin/mailcatcher:5:in `<top (required)>'
       C:/Ruby23-x64/bin/mailcatcher:22:in `load'
       C:/Ruby23-x64/bin/mailcatcher:22:in `<main>'
fnxnet commented 7 years ago

Same here on ruby 2.1.5p273 (2014-11-13) [x86_64-linux-gnu]


Exception: "\xC5" from ASCII-8BIT to UTF-8
    Backtrace:
       /usr/lib/ruby/2.1.0/json/common.rb:223:in `encode'
       /usr/lib/ruby/2.1.0/json/common.rb:223:in `generate'
       /usr/lib/ruby/2.1.0/json/common.rb:223:in `generate'
       mailcatcher (0.6.5) lib/mail_catcher/web/application.rb:69:in `block (3 levels) in <class:Application>'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `call'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block (3 levels) in push'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `each'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block (2 levels) in push'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `each'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `block in push'
       eventmachine (1.0.9.1) lib/eventmachine.rb:256:in `call'
       eventmachine (1.0.9.1) lib/eventmachine.rb:256:in `schedule'
       eventmachine (1.0.9.1) lib/em/channel.rb:43:in `push'
       mailcatcher (0.6.5) lib/mail_catcher/mail.rb:57:in `block in add_message'
       eventmachine (1.0.9.1) lib/eventmachine.rb:981:in `call'
       eventmachine (1.0.9.1) lib/eventmachine.rb:981:in `block in run_deferred_callbacks'
       eventmachine (1.0.9.1) lib/eventmachine.rb:978:in `times'
       eventmachine (1.0.9.1) lib/eventmachine.rb:978:in `run_deferred_callbacks'
       eventmachine (1.0.9.1) lib/eventmachine.rb:193:in `run_machine'
       eventmachine (1.0.9.1) lib/eventmachine.rb:193:in `run'
       mailcatcher (0.6.5) lib/mail_catcher.rb:173:in `run!'
       mailcatcher (0.6.5) bin/mailcatcher:5:in `<top (required)>'
       /usr/local/bin/mailcatcher:23:in `load'
       /usr/local/bin/mailcatcher:23:in `<main>'
    Please submit this as an issue at http://github.com/sj26/mailcatcher/issues
znz commented 7 years ago

I think it is bug of mail sender, not mailcatcher. see https://github.com/sj26/mailcatcher/issues/322

kdiogenes commented 6 years ago

I use UTF-8 in from, like the following code in ActionMailer::Base:

default from: 'leão - my latin america brand <no-reply@leao.com.br>'

I did the following commit only to show a possible fix https://github.com/cerdiogenes/mailcatcher/commit/b2520fff2f2e53079818847877a2227f924a1c19

What do you think about it @sj26? If you think it's acceptable I can prepare a PR.

Best regards!

znz commented 6 years ago

I think receiving mails with 8bit is root of problems. Because SMTP server of mailcatcher does not respond with 8BITMIME to the EHLO command, it should reject mail with 8bit from bad mail clients. I think sane mail clients re-send mail with 7bit encoding. It can help to reduce mail clients that violate RFCs.

kdiogenes commented 6 years ago

@znz my email body is encoded with 7bit (I don't do anything special to have it, ActionMailer already send it ok), but the from header field keeps giving error, this is my from header:

From: =?UTF-8?Q?AlfaCon_-_AlfaCon_Concursos_P=C3=BAblicos_<no-reply@alf?=
 =?UTF-8?Q?a-dev.com.br:3000>?=

The body have the proper charset and encoding:

Content-Type: multipart/mixed;
 boundary="--==_mimepart_5b61a61b9d9a5_159c2b07c5d3d950208f3";
 charset=UTF-8
Content-Transfer-Encoding: 7bit

----==_mimepart_5b61a61b9d9a5_159c2b07c5d3d950208f3
Content-Type: text/html;
 charset=UTF-8
Content-Transfer-Encoding: quoted-printable

And in the body I have this string <p>Obrigado por comprar no AlfaCon Concursos P=C3=BAblicos</p>

If I remove the accent from the from header I get no exception and mailcatcher show the email properly.

So, my question is basically how I can use accents in from field?

djomlastic commented 5 years ago

I'm not a ruby dev, but I believe this is related. Using mailcatcher in a vagrant box that serves as a magento development server. If email contains some non-ascii characters (german umlauts, euro symbol), it doesn't get sent. Apache error log:

/var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/message.rb:2032:in `match': invalid byte sequence in US-ASCII (ArgumentError)
    from /var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/message.rb:2032:in `match'
    from /var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/message.rb:2032:in `set_envelope_header'
    from /var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/message.rb:2132:in `init_with_string'
    from /var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/message.rb:137:in `initialize'
    from /var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/mail.rb:51:in `new'
    from /var/lib/gems/2.5.0/gems/mail-2.7.1/lib/mail/mail.rb:51:in `new'
    from /var/lib/gems/2.5.0/gems/mailcatcher-0.7.1/bin/catchmail:59:in `<top (required)>'
    from /usr/local/bin/catchmail:23:in `load'
    from /usr/local/bin/catchmail:23:in `<main>

So match function's argument is US-ASCII encoded, which is weird. Ruby / encoding related stuff:

vagrant@syone:~$ ruby -v
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]
vagrant@syone:~$ ruby -e 'p Encoding.default_external'
#<Encoding:UTF-8>
vagrant@syone:~$ locale
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC=C.UTF-8
LC_TIME=en_US.UTF-8
LC_COLLATE="C.UTF-8"
LC_MONETARY=C.UTF-8
LC_MESSAGES="C.UTF-8"
LC_PAPER=C.UTF-8
LC_NAME=C.UTF-8
LC_ADDRESS=C.UTF-8
LC_TELEPHONE=C.UTF-8
LC_MEASUREMENT=C.UTF-8
LC_IDENTIFICATION=C.UTF-8
LC_ALL=

But if I print out Encoding.default_external from catchmail , it really uses US-ASCII. Setting Encoding.default_external = en_US.UTF-8 within catchmail solves the problem, also changing line 59 within catchmail to message = Mail.new($stdin.read.force_encoding('UTF-8')) solves it. So I ended up changing sendmail_path within php.ini to: sendmail_path = "LANG=en_US.UTF-8 /usr/bin/env catchmail -f vagrant@syone" That works, but I still don't understand why catchmail thinks external encoding is US-ASCII.

sj26 commented 3 years ago

MailCatcher does not advertise itself as 8-bit capable, and so mail clients should be sending mail in ascii compatible encodings. A feature request for 8 bit transfer encodings is now here, please follow if you're interested: #445