NoMethodError: undefined method `parameters' for #<Mail::UnstructuredField

net1957 commented 14 years ago

Mail 2.2.0 choke on in class ContentTypeElement with : ContentTypeElement can not parse |Multipart/Mixed| Reason was: Expected one of text, image, audio, video, application, 7bit, 8bit, binary, quoted-printable, base64, message, multipart at line 1, column 1 (byte 1) after

on this header : Content-type: Multipart/Mixed; boundary=Message-Boundary-27177

on ruby 1.9.1p378 (2010-01-10 revision 26273) [i386-mingw32]

header are : Received: from spooler by sk-targa.sk.loc (Mercury/32 v4.72); 25 Apr 2010 21:13:07 +0200 X-Envelope-To: serge.kk@sk.loc X-CLAMWALL: Passed through antiviral test by ClamWall 1.4.0.96 on sk-targa.sk.loc (910) Return-path: serge.kk@sk.loc Received: from 192.168.0.2 by sk-targa.sk.loc (Mercury/32 v4.72) with ESMTP ID MG0002E1; 25 Apr 2010 21:12:57 +0200 From: "KUENY Serge" serge.kk@sk.loc To: serge.kk@sk.loc Date: Sun, 25 Apr 2010 21:12:57 +0200 MIME-Version: 1.0 Content-type: Multipart/Mixed; boundary=Message-Boundary-27177 Subject: pop3 Reply-to: serge.kk@sk.loc Message-ID: 4BD49439.30394.F8C705E@serge.kk@sk.loc Priority: normal X-mailer: Pegasus Mail for Windows (4.52, FR v4.51 R1.0) X-Text-Classification: normal X-POPFile-Link: http://127.0.0.1:8090/jump_to_message?view=23540 X-Recipient: serge.kk@sk.loc

lls commented 14 years ago

I have found the same error too in some mails. In my case they had a wrong content_type (images/jpeg instead of image/jpeg) but it would be cool that it just skip the part instead of failing. Its easy to reproduce (using 2.2.1):

require 'rubygems'
require 'mail'

Mail::VERSION::STRING

mail = Mail.new do
      to "aaaa@aaaa.aaa"
      from "aaaa2@aaaa.aaa"
      subject "a subject"
      date Time.now
      text_part do
    content_type 'text/plain; charset=UTF-8'
    body "a body\nsimple\n"
      end
    end

mail.attachments["tux.jpg"] = { :content => File.read("/tmp/tux.jpg")}

puts "no problem here"
Mail.new mail.encoded

puts "fails_here"
Mail.new mail.encoded.gsub("image/jpeg",  "images/jpeg")

the result is: irb irb(main):001:0> require 'rubygems' => true irb(main):002:0> require 'mail' => true irb(main):003:0> irb(main):004:0* Mail::VERSION::STRING => "2.2.1" irb(main):005:0> irb(main):006:0* mail = Mail.new do irb(main):007:1* to "aaaa@aaaa.aaa" irb(main):008:1> from "aaaa2@aaaa.aaa" irb(main):009:1> subject "a subject" irb(main):010:1> date Time.now irb(main):011:1> text_part do irb(main):012:2* content_type 'text/plain; charset=UTF-8' irb(main):013:2> body "a body\nsimple\n" irb(main):014:2> end irb(main):015:1> end => #<Mail::Message:69994379933780, Multipart: true, Headers: <Date: Thu, 13 May 2010 12:51:29 +0200>, <From: aaaa2@aaaa.aaa>, <To: aaaa@aaaa.aaa>, , <Content-Type: multipart/mixed>> irb(main):016:0> irb(main):017:0* mail.attachments["tux.jpg"] = { :content => File.read("/tmp/tux.jpg")} => {} irb(main):018:0> irb(main):019:0* puts "no problem here" no problem here => nil irb(main):020:0> Mail.new mail.encoded => #<Mail::Message:69994379132740, Multipart: true, Headers: <Date: Thu, 13 May 2010 12:51:29 +0200>, <From: aaaa2@aaaa.aaa>, <To: aaaa@aaaa.aaa>, <Message-ID: 4bebd9b199725_118f3fa8d8a7a9ac691ca@mypc.mail>, , <Mime-Version: 1.0>, <Content-Type: multipart/mixed; boundary="--==_mimepart_4bebd9b1929f6_118f3fa8d8a7a9ac688b8"; charset=UTF-8>, <Content-Transfer-Encoding: 7bit>> irb(main):021:0> irb(main):022:0* puts "fails_here" fails_here => nil irb(main):023:0> Mail.new mail.encoded.gsub("image/jpeg", "images/jpeg") NoMethodError: undefined method parameters' for #<Mail::UnstructuredField:0x7f51ac1e67c0> from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/field.rb:122:insend' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/field.rb:122:in method_missing' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/header.rb:164:incharset' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/header.rb:78:in fields=' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/header.rb:76:ineach' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/header.rb:76:in fields=' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/header.rb:250:insplit_header' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/header.rb:40:in initialize' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/message.rb:359:innew' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/message.rb:359:in header=' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/part.rb:88:inparse_message' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/message.rb:1839:in init_with_string' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/message.rb:119:ininitialize' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/body.rb:260:in new' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/body.rb:260:insplit!' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/body.rb:260:in each' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/body.rb:260:insplit!' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/message.rb:1756:in separate_parts' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/message.rb:1840:ininit_with_string' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/message.rb:119:in initialize' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/mail.rb:50:innew' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/mail.rb:50:in `new' from (irb):23 from :0irb(main):024:0>

lls commented 14 years ago

I have been realized that problem is quite worse than I thought: I have been testing 2.2.0/1 and many mails that were parsed in 2.1.3 are not parsed now. So, I can't use 2.2 version since it is broken for my app.

This is a sample that shows how it works for older version and not for new one:

Let's try this code and setting Mail version to 2.1.3: require 'rubygems' gem 'mail', '<= 2.1.3' require 'mail'

puts Mail::VERSION::STRING

Mail.new "To:aa@aa.aa\nFrom: bb@bb.bb\nSubject: a subject\nContent-Type: text/plain; charset=us-ascii\n\nsome text\n\n"
Mail.new "To:aa@aa.aa\nFrom: bb@bb.bb\nSubject: a subject\nContent-Type: Text/Plain\n\nsome text\n\n"
Mail.new "To:aa@aa.aa\nFrom: bb@bb.bb\nSubject: a subject\nContent-Type: Text/Plain; charset=us-ascii\n\nsome text\n\n"

all mails are parsed fine: irb irb(main):001:0> require 'rubygems' => true irb(main):002:0> gem 'mail', '<= 2.1.3' => true irb(main):003:0> require 'mail' => true irb(main):004:0> irb(main):005:0* puts Mail::VERSION::STRING 2.1.3 => nil irb(main):006:0> irb(main):007:0* Mail.new "To:aa@aa.aa\nFrom: bb@bb.bb\nSubject: a subject\nContent-Type: text/plain; charset=us-ascii\n\nsome text\n\n" => #<Mail::Message:69823812939580, Multipart: false, Headers: <From: bb@bb.bb>, <To: aa@aa.aa>, , <Content-Type: text/plain; charset=us-ascii>> irb(main):008:0> Mail.new "To:aa@aa.aa\nFrom: bb@bb.bb\nSubject: a subject\nContent-Type: Text/Plain\n\nsome text\n\n" => #<Mail::Message:69823812860680, Multipart: false, Headers: <From: bb@bb.bb>, <To: aa@aa.aa>, , <Content-Type: Text/Plain>> irb(main):009:0> Mail.new "To:aa@aa.aa\nFrom: bb@bb.bb\nSubject: a subject\nContent-Type: Text/Plain; charset=us-ascii\n\nsome text\n\n" => #<Mail::Message:69823812785500, Multipart: false, Headers: <From: bb@bb.bb>, <To: aa@aa.aa>, , <Content-Type: Text/Plain; charset=us-ascii>>

Now lets start a new irb and try with 2.2.1 version:

require 'rubygems'
gem 'mail', '>= 2.2.0'
require 'mail'

puts Mail::VERSION::STRING

Mail.new "To:aa@aa.aa\nFrom: bb@bb.bb\nSubject: a subject\nContent-Type: text/plain; charset=us-ascii\n\nsome text\n\n"
Mail.new "To:aa@aa.aa\nFrom: bb@bb.bb\nSubject: a subject\nContent-Type: Text/Plain\n\nsome text\n\n"
Mail.new "To:aa@aa.aa\nFrom: bb@bb.bb\nSubject: a subject\nContent-Type: Text/Plain; charset=us-ascii\n\nsome text\n\n"

As we can see, last mail fails parsing: irb irb(main):001:0> require 'rubygems' => true irb(main):002:0> gem 'mail', '>= 2.2.0' => true irb(main):003:0> require 'mail' => true irb(main):004:0> irb(main):005:0* puts Mail::VERSION::STRING 2.2.1 => nil irb(main):006:0> irb(main):007:0* Mail.new "To:aa@aa.aa\nFrom: bb@bb.bb\nSubject: a subject\nContent-Type: text/plain; charset=us-ascii\n\nsome text\n\n" => #<Mail::Message:70079067537860, Multipart: false, Headers: <From: bb@bb.bb>, <To: aa@aa.aa>, , <Content-Type: text/plain; charset=us-ascii>> irb(main):008:0> Mail.new "To:aa@aa.aa\nFrom: bb@bb.bb\nSubject: a subject\nContent-Type: Text/Plain\n\nsome text\n\n" => #<Mail::Message:70079067160400, Multipart: false, Headers: <From: bb@bb.bb>, <To: aa@aa.aa>, , <Content-Type: Text/Plain>> irb(main):009:0> Mail.new "To:aa@aa.aa\nFrom: bb@bb.bb\nSubject: a subject\nContent-Type: Text/Plain; charset=us-ascii\n\nsome text\n\n" NoMethodError: undefined method main_type' for #<Mail::UnstructuredField:0x7f791bcae6d0> from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/field.rb:122:insend' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/field.rb:122:in method_missing' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/message.rb:1400:inmain_type' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/message.rb:1421:in multipart?' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/message.rb:1840:ininit_with_string' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/message.rb:119:in initialize' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/mail.rb:50:innew' from /var/lib/gems/1.8/gems/mail-2.2.1/lib/mail/mail.rb:50:in `new' from (irb):9 from :0

Looks like caps in the content_type with some charset breaks parsing. As a curious side note, many mails that had failed in my tests where from ruby-core mail list :p

taskjuggler commented 14 years ago

Same problem for me. mail 2.2.x breaks on many real-world mails that 2.1.3 processed fine. This is a show stopper for me. I cannot upgrade to 2.2.x unless this bug has been fixed.

Here is another example that shows the problem:

require 'mail' --8<-- Mail.new <<'EOT' From joe@company.com Fri Feb 19 08:41:30 2010 From: Big Bug bb@bug.com To: rubymail@ruby-lang.org Subject: undef method parameter bug Date: Fri, 19 Feb 2010 10:08:29 +0300 MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Message-Id: 201002191008.30117.foo.bar@company.com

foo bar EOT -->8--

And this is the stack trace:

/usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/field.rb:122:in method_missing': undefined methodparameters' for Text/Plain; charset="iso-8859-1":Mail::UnstructuredField (NoMethodError)
from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/header.rb:164:in charset' from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/header.rb:78:inblock in fields=' from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/header.rb:76:in each' from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/header.rb:76:infields=' from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/header.rb:250:in split_header' from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/header.rb:40:ininitialize' from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/message.rb:359:in new' from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/message.rb:359:inheader=' from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/message.rb:1740:in parse_message' from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/message.rb:1839:ininit_with_string' from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/message.rb:119:in initialize' from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/mail.rb:50:innew' from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.1/lib/mail/mail.rb:50:in new' from mail-undef-par-meth-bug.rb:4:in

'

justinperkins commented 14 years ago

This is happening because the Mail gem assumes that the headers of an email are split on the carriage return, but many emails we receive have header values which span multiple lines.

The Mail::Header#split_header method is the source of this bug.

justinperkins commented 14 years ago

For clarity, I created issue #68 for header parsing bug.

justinperkins commented 14 years ago

Sorry guys for the ticket spam, I was wrong in my original thoughts on this bug. Not sure what is causing the underlying issue but some of the parameter/content-type parsing methods are getting choked up on certain emails.

lls commented 14 years ago

I did a little modification (http://github.com/lls/mail/blob/master/lib/mail/fields/content_type_field.rb), basically downcasing content type in case of error and it seems to work. There will probably be a better way to do it, maybe touching the parser, but I'm quite unfamiliar with it.

Hopefully mikel will look at this issue sometime. Unfortunately, he may also have a life and a job so he can't be fulltime in this project :P

justinperkins commented 14 years ago

I put in a similar fix last night as well. Basically the parameters method was getting invoked without checking if that object had the method (your fix addresses this as well). There were a few other places that had a similar issue which popped up once I fixed the parameters one, so there's a fix in my commit for those as well.

See here: http://github.com/justinperkins/mail/commit/5c64e85aaca0798ee275097d8d031e46010fcf0f

mikel commented 14 years ago

OK, so lets get this fixed :) I am at RailsConf now and playing with releasing another version of mail and putting in some of the more recent fixes.

Justin, please do write a spec, the easiest spec to write would be just to find an email that fails with mail, and creating a new file in the spec/fixtures/emails/error_emails/ directory. Just give it an intelligent sounding filename so it is clear the error it makes.

Then you should be able to run the specs, see it fail on that email, and then applying the fixes should handle it.

If you can incorporate Ils and your fixes to a new commit, with that failing spec, that would be awesome :)

Mikel

justinperkins commented 14 years ago

Well, as I said earlier, I am not a spec guy. I tried my best, installed rspec and the seemingly endless chain of dependency gems, and then ran spec *_/__spec.rb from the spec directory and oh boy did I get a lot of failures. 89 to be exact. It's probably a versioning issue or something, but it's really hard for me to debug since I am in unfamiliar territory. I have an email on hand that triggers this exception and have verified that it is fixed with the latest copy of my fork. I'm sorry I cannot be more help.

I will checkin the email I was referring to to my fork and send you a pull request.

libc commented 14 years ago

http://gist.github.com/467892 fixes this issue.

I'm not happy with [iI] [mM] [aA] [gG] [eE] though, didn't found other way to ignore case with treetop.

The piece of header in spec taken from real-life e-mail.

mcculloughsean commented 14 years ago

I applied libc's patch, but it still fails on certain emails.

Check issue 82 for an example.

libc commented 14 years ago

Are you sure you applied it right?

  >> Mail.new(open('http://gist.github.com/raw/472080/11dc9c739e115e430b3385ff3abb67c22fcb159b/test.image-att.eml').read).parts[1].content_type
 => "image/jpeg; name=IMG_8476.jpg"

Update: I see, content-disposition issue.

joshgoebel commented 14 years ago

I added a spec that shows this problem and issued a pull request.

http://github.com/yyyc514/mail/commit/3a256a72aae7ddcea348f772dbbad75e0c18b368

joshgoebel commented 14 years ago

Added some commit notes here:

http://github.com/justinperkins/mail/commit/5c64e85aaca0798ee275097d8d031e46010fcf0f#commitcomment-113837

This isn't a 100% fix for the underlying problem.

joshgoebel commented 14 years ago

Just down case the beginning of the Content-Type header inside the cleaned methods before it's passed to the parser. Makes sense to me since the parser seems to only expect downcased values in the first place, and that's how you usually see content type written out in "proper" form.

joshgoebel commented 14 years ago

Ok, libc's fix looks good. I've committed that to my fork and issues a pull request.

http://github.com/yyyc514/mail/commit/510cf0d35968ecd0acbe121b11e44d165f28095a

maccman commented 14 years ago

I've also come across this problem. Looking forward to when it's in core.

joshgoebel commented 14 years ago

If you use Bundler it's easy enough to pin to an arbitrary version on github... here is how I've done it:

gem "mail", :git => "git://github.com/yyyc514/mail.git", :tag => "2.2.5-content_type_fix"

I've merged in libc's patch into my fork and then tagged it... and point bundler to that... and if you aren't using Bundler you should be. :)

kuahyeow commented 13 years ago

+1 to yyyc514 's root cause. For some reason the first letter of the Content-Type header has to be lower-case. I have applied http://github.com/yyyc514/mail/commit/2139c003b924c87c9ae2062decff5ea2f5ccc0ce to my project but libc's fix is okay too. http://github.com/yyyc514/mail/commit/510cf0d35968ecd0acbe121b11e44d165f28095a

e.g. This causes the error in the title


Content-Type: Text/HTML;

This has no errors.


Content-Type: text/HTML;