packs/email imap-sensor not handling multipart mime messages

[nido009]

the email pack has a problem with multipart mime messages. upon hitting a multipart message it crashes with "Can't decorde message body: DecodeError(TypeError(......"

_process_message() already parses mime for the headers. instead of body = message.body it should be something like "body=mime_msg.body". then it would only be parsed if not multipart.

[Kami]

Thanks for the report.

Can you please provide more details - gisting a whole raw email message would be great (you can of course mask / remove sensitive parts).

When I worked on the support for attachments, I didn't encounter this issue (I tested it with multiple different attachments) so I'm curious what is causing this issue. Or maybe this only affects message with multiple body parts (I also tested an email with two parts - raw and html and it worked fine).

[nido009]

still with the multipart fix i receive the following errors when a mail is received via imap:

==> st2sensorcontainer.log <== Jul 29 13:57:53 localhost st2sensorcontainer[18719]: DEBUG 140030889148080 log [-] process_task Jul 29 13:57:53 localhost st2sensorcontainer[18719]: DEBUG 140030889148080 log [-] body: TriggerDB(description=None, id=55b8d8d39c99383f8d82a834, name="st2.sensor.process_exit", pack="core", parameters={}, type="core.st2.sensor.process_exit") Jul 29 13:57:53 localhost st2sensorcontainer[18719]: DEBUG 140030889148080 log [-] message.properties: {'priority': 0, 'application_headers': {}, 'delivery_mode': 2, 'content_encoding': u'binary', 'content_type': u'application/x-python-serialize'} Jul 29 13:57:53 localhost st2sensorcontainer[18719]: DEBUG 140030889148080 log [-] message.delivery_info: {'consumer_tag': u'1', 'redelivered': False, 'routing_key': u'update', 'delivery_tag': 8, 'exchange': u'st2.trigger'} Jul 29 13:57:53 localhost st2sensorcontainer[18719]: DEBUG 140030889148080 log [-] Skipping message <kombu.transport.pyamqp.Message object at 0x7f5b7ff73348> since't trigger_type doesn't match (type=core.st2.sensor.process_exit)

Jul 29 13:58:22 localhost st2sensorcontainer[18720]: DEBUG 140505277216720 log [-] [IMAPSensor]: entering poll Jul 29 13:58:22 localhost st2sensorcontainer[18720]: DEBUG 140505277216720 log [-] [IMAPSensor]: polling mailbox dummyhost_main Jul 29 13:58:22 localhost st2sensorcontainer[18720]: DEBUG 140505277216720 log [-] [IMAPSensor]: Processing 1 new messages Jul 29 13:58:22 localhost st2sensorcontainer[18720]: DEBUG 140505277216720 log [-] Dispatching trigger (trigger=email.imap.message,payload={'trigger': 'email.imap.message', 'payload': {'body': [u'test test\r\n\r\nnido009\r\n', u'

test test

nido009

\r\n

\r\n'], 'to': u'st2@dummyhost.de', 'from': u'Nils Dohse n.dohse@gmail.com', 'uid': 8, 'headers': [('Return-Path', u'n.dohse@gmail.com'), ('X-Original-To', u'st2@dummyhost.de'), ('Delivered-To', u'st2@dummyhost.de'), ('Received', u'from mail-lb0-f173.google.com (mail-lb0-f173.google.com [209.85.217.173])\tby pl2.addix.net (Postfix) with ESMTPS id 299CF40E0A\tfor st2@dummyhost.de; Wed, 29 Jul 2015 15:57:58 +0200 (CEST)'), ('Received', u'by lbbyj8 with SMTP id yj8so7522681lbb.0 for st2@dummyhost.de; Wed, 29 Jul 2015 06:57:57 -0700 (PDT)'), ('Dkim-Signature', u'v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=b16XL/kDFetWKcpKzL678LWI+Y6zff+okEBQ5SAagm8=; b=USdFsbg7bkvf/OVaQFfdPhnZW/IZlsmf6gdjhnmcibCX6GHfYIP+XL52I3Wgv3skXh 4qCZ1PUfBpx0WbbcJLf4ephOZ2CtTOOo1X18VbXEMtPMgyK/vWQgBn95N5ok67RaZF9O z4Q5+PuSvDgXEqiQ12ysfc9LxDvZw+oG0Jb/Jd+ED4e8Dq/mdP4rUSyj49XUHcKjZK1D 622cRLj9cINkDz7EAZ43M3imwmWzEoggbDVzV0H0wS33NA/b0fU8XXN7JziVbPs0fdOb dsGFKW05Db2q4DwxVMy9k2wrcT4Q0Aa3hbwcXduMaA0HTYKuF+WQF863VJ7FlpjaIoui +vpA=='), ('Mime-Version', u'1.0'), ('X-Received', u'by 10.152.21.132 with SMTP id v4mr38582531lae.18.1438178277638; Wed, 29 Jul 2015 06:57:57 -0700 (PDT)'), ('Received', u'by 10.114.186.33 with HTTP; Wed, 29 Jul 2015 06:57:57 -0700 (PDT)'), ('Date', u'Wed, 29 Jul 2015 15:57:57 +0200'), ('Message-Id', u'CANBhjP9CMYQM2jE88peDnP0MyaCA_g3YdVM6CAkih4o-R4SVyA@mail.gmail.com'), ('Subject', u'test-29.2'), ('From', u'Nils Dohse n.dohse@gmail.com'), ('To', u'st2@dummyhost.de'), ('Content-Type', ContentType('multipart', 'alternative', {'boundary': u'089e0158b6c85ccabc051c03f9ee'}))], 'date': u'Wed, 29 Jul 2015 15:57:57 +0200', 'has_attachments': False, 'attachments': [], 'message_id': u'CANBhjP9CMYQM2jE88peDnP0MyaCA_g3YdVM6CAkih4o-R4SVyA@mail.gmail.com', 'subject': u'test-29.2'}}) Jul 29 13:58:22 localhost st2sensorcontainer[18720]: DEBUG 140505277216720 connection [-] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2015 Pivotal Software, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'per_consumer_qos': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'cluster_name': u'rabbit@st2express', u'platform': u'Erlang/OTP', u'version': u'3.5.4'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US'] Jul 29 13:58:22 localhost st2sensorcontainer[18720]: DEBUG 140505277216720 connection [-] Open OK! Jul 29 13:58:22 localhost st2sensorcontainer[18720]: DEBUG 140505277216720 channel [-] using channel_id: 1 Jul 29 13:58:22 localhost st2sensorcontainer[18720]: DEBUG 140505277216720 channel [-] Channel open

==> st2rulesengine.log <== Jul 29 13:58:22 localhost st2rulesengine[18520]: ERROR 140579278949264 mixins [-] Can't decode message body: DecodeError(ImportError('No module named flanker.mime.message.headers.wrappers',),) (type:u'application/x-python-serialize' encoding:u'binary' raw:'\'\x80\x02}q\x01(U\x07triggerq\x02U\x12email.imap.messageq\x03U\x07payloadq\x04}q\x05(U\x04bodyq\x06]q\x07(X\x16\x00\x00\x00test test\r\n\r\nnido009\r\nq\x08X\xce\x00\x00\x00

test test

nido009

\r\n

\r\nq\teU\x02toq\nX\x10\x00\x00\x00st2@dummyhost.deq\x0bU\x04fromq\x0cX\x1e\x00\x00\x00Nils Dohse n.dohse@gmail.comq\rU\x03uidq\x0eK\x08U\x07headersq\x0f]q\x10(U\x0bReturn-Pathq\x11X\x13\x00\x00\x00n.dohse@gmail.com\x86q\x12U\rX-Original-Toq\x13X\x10\x00\x00\x00st2@dummyhost.de\x86q\x14U\x0cDelivered-Toq\x15X\x10\x00\x00\x00st2@dummyhost.de\x86q\x16U\x08Receivedq\x17X\xbd\x00\x00\x00from mail-lb0-f173.google.com (mail-lb0-f173.google.com [209.85.217.173])\tby pl2.addix.net (Postfix) with ESMTPS id 299CF40E0A\tfor st2@dummyhost.de; Wed, 29 Jul 2015 15:57:58 +0200 (CEST)\x86q\x18U\x08Receivedq\x19Xm\x00\x00\x00by lbbyj8 with SMTP id yj8so7522681lbb.0 for st2@dummyhost.de; Wed, 29 Jul 2015 06:57:57 -0700 (PDT)\x86q\x1aU\x0eDkim-Signatureq\x1bXP\x02\x00\x00v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=b16XL/kDFetWKcpKzL678LWI+Y6zff+okEBQ5SAagm8=; b=USdFsbg7bkvf/OVaQFfdPhnZW/IZlsmf6gdjhnmcibCX6GHfYIP+XL52I3Wgv3skXh 4qCZ1PUfBpx0WbbcJLf4ephOZ2CtTOOo1X18VbXEMtPMgyK/vWQgBn95N5ok67RaZF9O z4Q5+PuSvDgXEqiQ12ysfc9LxDvZw+oG0Jb/Jd+ED4e8Dq/mdP4rUSyj49XUHcKjZK1D 622cRLj9cINkDz7EAZ43M3imwmWzEoggbDVzV0H0wS33NA/b0fU8XXN7JziVbPs0fdOb dsGFKW05Db2q4DwxVMy9k2wrcT4Q0Aa3hbwcXduMaA0HTYKuF+WQF863VJ7FlpjaIoui +vpA==\x86q\x1cU\x0cMime-Versionq\x1dX\x03\x00\x00\x001.0\x86q\x1eU\nX-Receivedq\x1fXe\x00\x00\x00by 10.152.21.132 with SMTP id v4mr38582531lae.18.1438178277638; Wed, 29 Jul 2015 06:57:57 -0700 (PDT)\x86q U\x08Receivedq!XA\x00\x00\x00by 10.114.186.33 with HTTP; Wed, 29 Jul 2015 06:57:57 -0700 (PDT)\x86q"U\x04Dateq#X\x1f\x00\x00\x00Wed, 29 Jul 2015 15:57:57 +0200\x86q$U\nMessage-Idq%XD\x00\x00\x00CANBhjP9CMYQM2jE88peDnP0MyaCA_g3YdVM6CAkih4o-R4SVyA@mail.gmail.com\x86q&U\x07Subjectq\'X\t\x00\x00\x00test-29.2\x86q(U\x04Fromq)X\x1e\x00\x00\x00Nils Dohse n.dohse@gmail.com\x86q*U\x02Toq+X\x10\x00\x00\x00st2@dummyhost.de\x86q,U\x0cContent-Typeq-cflanker.mime.message.headers.wrappers\nContentType\nq.U\x15multipart/alternativeq/}q0U\x08boundaryX\x1c\x00\x00\x00089e0158b6c85ccabc051c03f9ees\x86q1\x85\x81q2}q3(U\x04mainq4U\tmultipartU\x03subq5U\x0balternativeub\x86q6eU\x04dateq7X\x1f\x00\x00\x00Wed, 29 Jul 2015 15:57:57 +0200q8U\x0fhas_attachmentsq9\x89U\x0battachmentsq:]U\nmessage_idq;XD\x00\x00\x00CANBhjP9CMYQM2jE88peDnP0MyaCA_g3YdVM6CAkih4o-R4SVyA@mail.gmail.comq<U\x07subjectq=X\t\x00\x00\x00test-29.2q>uu.\''')

==> st2sensorcontainer.log <== Jul 29 13:58:22 localhost st2sensorcontainer[18720]: DEBUG 140505277216720 channel [-] Closed channel #1

[nido009]

as far as i can see this error is not related to the email pack directly, but instead in the rulesengine, cause the flanker include is missing there.

[Kami]

This is rather weird.

Rules engine shouldn't import anything from there. We use pickle serializer for message bus messages, so it could be something weird going on there (e.g. payload is some how being treated as Python code when de-seralizing it and this results in the import...).

Or it could simply be error message which occurs inside the sensor (sensor container), but it's being logged in a weird way inside the rules engine.

@nido009 Can you please forward me the email in question to (tomaz at stackstorm.com) and I will try to reproduce the issue as soon as I can.

[nido009]

@Kami you have mail.

[jfryman]

This issue is also affecting one of our POCs that we have ongoing.

[Kami]

@nido009 @jfryman Thanks. I will dig in today and try to reproduce the issue.

[Kami]

@nido009 I tried to reproduce this issue using the email you forwarded to me and yes, the issue doesn't lie in the sensor.

The sensor works just fine and the trigger is dispatched correctly. It looks like it's an issue with the rules engine. I will dig further.

[Kami]

Yep, I just confirmed it - the exception is throw inside the rules engine (kombu library) when we try to de-serialize (unpickle) the message on the message bus.

I'm looking into a fix now.