JMAConsulting / biz.jmaconsulting.mte

Mandrill Emails Extension for CiviCRM
10 stars 21 forks source link

Webhook failing because of IDS? #105

Closed dyguk closed 5 years ago

dyguk commented 7 years ago

Firstly, it needs to be said that the Mandrill extension is excellent! Well done to you all. It really provides excellent functionality.

Now, I'm having an issue that may be similar to issue #47, or not sure.

Mandrill's webhooks are reporting this: Error: POST to http://mywebsite.com/civicrm/ajax/mte/callback?mandrillSecret=(our secret key fr Administer > system settings > mandrill smtp settings) failed with 500:
(nothing following)

Then our Civicrm log is filling with the kinds of things below for many Mandrill webhook tries.

Do you have any clues about what I could do to remedy this? (Other than turning off the webhooks in Mandrill)

(I've taken out personal details, checksums and keys and replaced them with '(...)' )

Thanks again for this excellent piece of work!

Nov 06 14:35:13 [info] $IDS Detector Details = Array ( [0] => Array ( [name] => mandrill_events [value] => [{"event":"open","ts":1478383723,"user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.8 (KHTML, like Gecko)","user_agent_parsed":{"type":"Email Client","ua_family":"Apple Mail","ua_name":"Apple Mail","ua_version":null,"ua_url":"http://en.wikipedia.org/wiki/Apple_mail","ua_company":"Apple Inc.","ua_company_url":"http://www.apple.com/","ua_icon":"http://cdn.mandrill.com/img/email-client-icons/apple-mail.png","os_family":"OS X","os_name":"OS X","os_url":"http://www.apple.com/osx/","os_company":"Apple Computer, Inc.","os_company_url":"http://www.apple.com/","os_icon":"http://cdn.mandrill.com/img/email-client-icons/macosx.png","mobile":false},"ip":"84.92.93.107","location":{"country_short":"GB","country":"United Kingdom","region":"England","city":"Sheffield","latitude":53.3829689026,"longitude":-1.46589994431,"postal_code":"-","timezone":"+01:00"},"_id":"17ba549027b34ebdaf19f917b947f62b","msg":{"ts":1478383232,"_id":"17ba549027b34ebdaf19f917b947f62b","state":"sent","subject":"Your subscription confirmation","email":"(...)@fudev.co.nz","tags":[],"opens":[{"ts":1478383737,"ip":"84.92.93.107","ua":"OS X/OS X/Chrome/Chrome 54.0.2840.71","location":"England, GB"},{"ts":1478383723,"ip":"84.92.93.107","location":"England, GB","ua":"OS X/OS X/Apple Mail/Apple Mail"}],"clicks":[{"ts":1478383737,"url":"http://mywebsite.com/civicrm/mailing/confirm?reset=1&cid=17095&sid=453&h=ced286059bf4e99e","ip":"84.92.93.107","location":"England, GB","ua":"OS X/OS X/Chrome/Chrome 54.0.2840.71"}],"smtp_events":[],"resends":[],"_version":"FTm-UnqLtJt4m8ofhvVJ-g","sender":"hello@druworldwide.com","template":null}},{"event":"open","ts":1478383423,"user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.8 (KHTML, like Gecko)","user_agent_parsed":{"type":"Email Client","ua_family":"Apple Mail","ua_name":"Apple Mail","ua_version":null,"ua_url":"http://en.wikipedia.org/wiki/Apple_mail","ua_company":"Apple Inc.","ua_company_url":"http://www.apple.com/","ua_icon":"http://cdn.mandrill.com/img/email-client-icons/apple-mail.png","os_family":"OS X","os_name":"OS X","os_url":"http://www.apple.com/osx/","os_company":"Apple Computer, Inc.","os_company_url":"http://www.apple.com/","os_icon":"http://cdn.mandrill.com/img/email-client-icons/macosx.png","mobile":false},"ip":"84.92.93.107","location":{"country_short":"GB","country":"United Kingdom","region":"England","city":"Sheffield","latitude":53.3829689026,"longitude":-1.46589994431,"postal_code":"-","timezone":"+01:00"},"_id":"19ac1011d86e421a89332d24463e71a5","msg":{"ts":1478382729,"_id":"19ac1011d86e421a89332d24463e71a5","state":"sent","subject":"(...), your gifts from the Yoga Show","email":"(...)@dru.com.au","tags":[],"opens":[{"ts":1478382740,"ip":"84.92.93.107","location":"England, GB","ua":"OS X/OS X/Apple Mail/Apple Mail"},{"ts":1478382807,"ip":"84.92.93.107","location":"England, GB","ua":"OS X/OS X/Apple Mail/Apple Mail"},{"ts":1478383423,"ip":"84.92.93.107","location":"England, GB","ua":"OS X/OS X/Apple Mail/Apple Mail"}],"clicks":[],"smtp_events":[{"ts":1478382798,"type":"sent","diag":"250 OK id=1c38tQ-0004JI-KA","source_ip":"198.2.136.4","destination_ip":"110.34.54.37","size":73793},{"ts":1478382798,"type":"sent","diag":"250 OK id=1c38tQ-0004JI-KA","source_ip":"198.2.136.4","destination_ip":"110.34.54.37","size":73793}],"resends":[],"_version":"3iBKuz3YlFao_nbD36yQKg","sender":"(...)@druworldwide.com","template":null}},{"event":"open","ts":1478433209,"user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.8 (KHTML, like Gecko)","user_agent_parsed":{"type":"Email Client","ua_family":"Apple Mail","ua_name":"Apple Mail","ua_version":null,"ua_url":"http://en.wikipedia.org/wiki/Apple_mail","ua_company":"Apple Inc.","ua_company_url":"http://www.apple.com/","ua_icon":"http://cdn.mandrill.com/img/email-client-icons/apple-mail.png","os_family":"OS X","os_name":"OS X","os_url":"http://www.apple.com/osx/","os_company":"Apple Computer, Inc.","os_company_url":"http://www.apple.com/","os_icon":"http://cdn.mandrill.com/img/email-client-icons/macosx.png","mobile":false},"ip":"84.92.93.107","location":{"country_short":"GB","country":"United Kingdom","region":"England","city":"Sheffield","latitude":53.3829689026,"longitude":-1.46589994431,"postal_code":"-","timezone":"+01:00"},"_id":"9e47b55ed2914f6fb936cb75a81a8c5c","msg":{"ts":1478432048,"_id":"9e47b55ed2914f6fb936cb75a81a8c5c","state":"sent","subject":"(...)18, your 2nd test meditation success course email","email":"(...)wells1+18@gmail.com","tags":[],"opens":[{"ts":1478433216,"ip":"84.92.93.107","location":"England, GB","ua":"OS X/OS X/Apple Mail/Apple Mail"},{"ts":1478433132,"ip":"84.92.93.107","location":"England, GB","ua":"OS X/OS X/Apple Mail/Apple Mail"},{"ts":1478433209,"ip":"84.92.93.107","location":"England, GB","ua":"OS X/OS X/Apple Mail/Apple Mail"}],"clicks":[],"smtp_events":[{"ts":1478432047,"type":"sent","diag":"250 2.0.0 OK 1478432047 b67si3733286ybi.14 - gsmtp","source_ip":"198.2.136.4","destination_ip":"64.233.176.26","size":55046},{"ts":1478432047,"type":"sent","diag":"250 2.0.0 OK 1478432047 b67si3733286ybi.14 - gsmtp","source_ip":"198.2.136.4","destination_ip":"64.233.176.26","size":55046}],"resends":[],"_version":"Ck30pxkJZPsJRglbaGBtCQ","metadata":{"CiviCRM_Mandrill_id":"49355.m.94.6416.f7c43944aab9ed6e"},"sender":"webadmin@druworldwide.com","template":null}}] [page] => /civicrm/ajax/mte/callback?mandrillSecret=(...) [userid] => [session] => m7xcwg2_bLE30SHtmMmGrnPCn36XlOMiFNGEsoBqlZQ [ip] => my.ip.address [reaction] => 2 [impact] => 59 )

)

Nov 06 14:35:37 [info] $IDS Detector Details = Array ( [0] => Array ( [name] => mandrill_events [value] => [{"event":"open","ts":1478433222,"user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.8 (KHTML, like Gecko)","user_agent_parsed":{"type":"Email Client","ua_family":"Apple Mail","ua_name":"Apple Mail","ua_version":null,"ua_url":"http://en.wikipedia.org/wiki/Apple_mail","ua_company":"Apple Inc.","ua_company_url":"http://www.apple.com/","ua_icon":"http://cdn.mandrill.com/img/email-client-icons/apple-mail.png","os_family":"OS X","os_name":"OS X","os_url":"http://www.apple.com/osx/","os_company":"Apple Computer, Inc.","os_company_url":"http://www.apple.com/","os_icon":"http://cdn.mandrill.com/img/email-client-icons/macosx.png","mobile":false},"ip":"84.92.93.107","location":{"country_short":"GB","country":"United Kingdom","region":"England","city":"Sheffield","latitude":53.3829689026,"longitude":-1.46589994431,"postal_code":"-","timezone":"+01:00"},"_id":"85f9d00940194647ac4fabb3b3ca43ce","msg":{"ts":1478433126,"_id":"85f9d00940194647ac4fabb3b3ca43ce","state":"sent","subject":"(...)10, Your welcome to Meditation Success course","email":"(...)wells1+10@gmail.com","tags":[],"opens":[{"ts":1478433222,"ip":"84.92.93.107","location":"England, GB","ua":"OS X/OS X/Apple Mail/Apple Mail"}],"clicks":[],"smtp_events":[{"ts":1478433126,"type":"sent","diag":"250 2.0.0 OK 1478433126 r73si5645090ywg.50 - gsmtp","source_ip":"198.2.136.4","destination_ip":"74.125.138.26","size":55039},{"ts":1478433126,"type":"sent","diag":"250 2.0.0 OK 1478433126 r73si5645090ywg.50 - gsmtp","source_ip":"198.2.136.4","destination_ip":"74.125.138.26","size":55039}],"resends":[],"_version":"m1E-rhRBVCNI4Tnr71lTvg","metadata":{"CiviCRM_Mandrill_id":"49362.m.94.6419.d7894aeb384a4138"},"sender":"webadmin@druworldwide.com","template":null}}] [page] => /civicrm/ajax/mte/callback?mandrillSecret=(...) [userid] => [session] => Qg7wPvKkDUSdgoYHIyxuE-jEdyT3oIIpcdZ5i8p_c6Y [ip] => my.ip.address [reaction] => 0 [impact] => 47 )

)

Nov 06 14:39:14 [info] $IDS Detector Details = Array ( [0] => Array ( [name] => mandrill_events [value] => [{"event":"open","ts":1478433218,"user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.8 (KHTML, like Gecko)","user_agent_parsed":{"type":"Email Client","ua_family":"Apple Mail","ua_name":"Apple Mail","ua_version":null,"ua_url":"http://en.wikipedia.org/wiki/Apple_mail","ua_company":"Apple Inc.","ua_company_url":"http://www.apple.com/","ua_icon":"http://cdn.mandrill.com/img/email-client-icons/apple-mail.png","os_family":"OS X","os_name":"OS X","os_url":"http://www.apple.com/osx/","os_company":"Apple Computer, Inc.","os_company_url":"http://www.apple.com/","os_icon":"http://cdn.mandrill.com/img/email-client-icons/macosx.png","mobile":false},"ip":"84.92.93.107","location":{"country_short":"GB","country":"United Kingdom","region":"England","city":"Sheffield","latitude":53.3829689026,"longitude":-1.46589994431,"postal_code":"-","timezone":"+01:00"},"_id":"581d6d81ae2241b89b1355a7ec5a2ea1","msg":{"ts":1478432374,"_id":"581d6d81ae2241b89b1355a7ec5a2ea1","state":"sent","subject":"(...)18, 3rd Meditation success email","email":"(...)wells1+18@gmail.com","tags":[],"opens":[{"ts":1478433210,"ip":"84.92.93.107","location":"England, GB","ua":"OS X/OS X/Apple Mail/Apple Mail"},{"ts":1478433133,"ip":"84.92.93.107","location":"England, GB","ua":"OS X/OS X/Apple Mail/Apple Mail"},{"ts":1478433218,"ip":"84.92.93.107","location":"England, GB","ua":"OS X/OS X/Apple Mail/Apple Mail"}],"clicks":[],"smtp_events":[{"ts":1478432406,"type":"sent","diag":"250 2.0.0 OK 1478432406 q2si5607544ywc.307 - gsmtp","source_ip":"198.2.136.4","destination_ip":"74.125.196.26","size":55010},{"ts":1478432406,"type":"sent","diag":"250 2.0.0 OK 1478432406 q2si5607544ywc.307 - gsmtp","source_ip":"198.2.136.4","destination_ip":"74.125.196.26","size":55010}],"resends":[],"_version":"V9Xa4ThKJCaLAmTxFaG_Og","metadata":{"CiviCRM_Mandrill_id":"49359.m.94.6418.2241d1f4e79a0980"},"sender":"webadmin@druworldwide.com","template":null}}] [page] => /civicrm/ajax/mte/callback?mandrillSecret=(...) [userid] => [session] => s-L7C-uHWx6KwQD77F4qvvQIookcp2eOIlUDfSIJjVw [ip] => my.ip.address [reaction] => 0 [impact] => 47 )

)

dyguk commented 7 years ago

mmmm. I should have added that this occurs if I send a test webhook from Mandrill, as follows:

Trigger only on Send:

Nov 06 15:02:53 [info] $IDS Detector Details = Array ( [0] => Array ( [name] => mandrill_events [value] => [{"event":"send","msg":{"ts":1365109999,"subject":"This an example webhook message","email":"example.webhook@mandrillapp.com","sender":"example.sender@mandrillapp.com","tags":["webhook-example"],"opens":[],"clicks":[],"state":"sent","metadata":{"user_id":111},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa","_version":"exampleaaaaaaaaaaaaaaa"},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa","ts":1478444573},{"event":"send","msg":{"ts":1365109999,"subject":"This an example webhook message","email":"example.webhook@mandrillapp.com","sender":"example.sender@mandrillapp.com","tags":["webhook-example"],"opens":[],"clicks":[],"state":"sent","metadata":{"user_id":111},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa1","_version":"exampleaaaaaaaaaaaaaaa"},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa1","ts":1478444573}] [page] => /civicrm/ajax/mte/callback?mandrillSecret=(....) [userid] => [session] => mrhMwgRY44mgISiPYuMLXMCZz1y1IoJfQXbOeAdJz7s [ip] => 46.17.89.44 [reaction] => 0 [impact] => 45 )

)

christopherreay commented 7 years ago

Hi, Ive just been looking at the civicrm mail tool for mailgun. They have an explicit patch for exactly this IDS issue.

There is a link to the patch, and an explaination of how to use it here: https://github.com/teamsinger/uk.teamsinger.civicrm.mailgun

artfulrobot commented 5 years ago

@christopherreay if you use mailgun you may be interested in this https://github.com/artfulrobot/mailgunny which is up to date with their latest API and does not use core overrides.

artfulrobot commented 5 years ago

See PR 117 - that initial PR was a bit messy, sorry!