search

JMAConsulting / biz.jmaconsulting.mte

Mandrill Emails Extension for CiviCRM
10 stars 21 forks source link

Webhooks not updating CiviCRM #56

Closed robbrandt closed 9 years ago

robbrandt commented 9 years ago

Hi. I have configured the Mandrill extension to be used in a 4.3.4. In all cases I can see, it is functioning well inside of CiviCRM. I have sent some test messages to myself, and have tested the "open" and "click" tracking. It is properly being recorded in Mandrill, but not sent back to CiviCRM. The webhook I created matches exactly what's in the Mandrill settings in CiviCRM.

FWIW, I have two different CiviCRM installations with webhooks installed in this Mandrill account. Could that be backing a difference?

shawnholt commented 9 years ago

Check IDS (civicrm intrusion detection) and make sure it is disabled. Be careful, as there are a couple of ways to disable it (i think I made a note in the docs.) Frankly, I never got this module to work properly, but that was definitely an issue.

robbrandt commented 9 years ago

It seems IDS is a v4.4 and newer feature. This is a 4.3.4 installation.

shawnholt commented 9 years ago

I think it has been around quite a while - back to version 2.x or 3.x if I'm not mistaken. may have moved around.

robbrandt commented 9 years ago

I've disabled IDS in civicrm.settings.php but it hasn't solved the problem.

shawnholt commented 9 years ago

As I said, I never got the webhooks to work properly, but IDS and Apache Mod Security definitely were triggered. Once I got past that I added a statement in the code and verified that the it made it to the plugin, but the bounce processing still didn't work. You may try that to see if the webhook is getting through. you can add a watchdog line in biz.jmaconsulting.mte/CRM/Mte/Page/Callback.php like: watchdog('MTE', '

%print_r
', array('%print_r' => print_r( $reponse, TRUE)));

Please let me know if you figure it out ...

robbrandt commented 9 years ago

I don't have mod_security enabled at this point, and IDS isn't either, so I think it's something else. My understanding is that a new version is pending that will be compatible with CiviCRM 4.5, so I just might wait for that and hope that it works.

JoeMurray commented 9 years ago

We're shortly going to release the 4.5 version. But I'm not sure how you can get two CiviCRM instances to work with a single Mandrill account. Seems to me like it has to call back to one or the other, but isn't able to call back to both.

robbrandt commented 9 years ago

Thanks for the reply. I added two webhooks because I could; I really didn't think of consequences until it didn't work. I don't know anything about the Mandrill API - for all I know, it knows which API key was used and can associate webhooks with the key.

The "2nd" (actually the original) webhook is a demo for a client. I was going to create a new account for them if/when they approved CiviCRM as their app and delete their webhook from my account at that time. But I suppose since it's free it won't be a big deal to create a potentially throw-away Mandrill account for them and move it there now.

robbrandt commented 9 years ago

I removed the "demo" webhook, only my real one remains. Still not working. Outbound activity in Mandrill for my test email says 4 Opens, 1 Click (which is correct). The webhook log says 5 1-event batches that are all marked as Completed. But still not showing up in CiviCRM.

JoeMurray commented 9 years ago

Thanks robbrandt for checking.

Mandrill won't call the webhooks a second time after the reconfiguration. Did you / could you try sending a few new non-bulk emails now that Mandrill is set up for just one CiviCRM instance?

pradpnayak commented 9 years ago

Hi robbrandt

  1. Could you please navigate to the file specified in the link and provide me with a debug of the variable 'response' on line 41? I just need to know what Mandrill returns in your case (mandrill_events).
  2. Also, could you please tell me the method in which you are sending mails to Mandrill? Is it via a transactional mail (non-bulk mail) or bulk email?
  3. Another thing I would like to know is the method in which your are verifying the clicks, opens and bounces. Are you doing it via the UI or at a database level?
robbrandt commented 9 years ago

Joe, that is what I was testing - new individual mails sent via Mandrill.

robbrandt commented 9 years ago

Pradeep, 1) You might need to be more specific. The link is http://crm.domain.com/civicrm/ajax/mte/callback?mandrillSecret=xxx which I think should mean the file is /sites/default/civicrm_extensions/biz.jmaconsulting.mte/CRM/Mte/Page/callback.php. When I add something like echo 'response: ',var_dump($reponse); to it I get nothing back when I browse to the URL above. So perhaps it's dying before it gets there.

2) I am using Mandrill both for bulk and non-bulk. Bulk is working fine. I am trying to get non-bulk working. I have filled in the new Mandril Smtp Settings - Outbound Mail found at /civicrm/mte/smtp?reset=1, and mail is definitely being sent.

3) I'm not sure what you mean. The clicks and opens are showing up in the Mandrillapp.com UI, but they are not showing up in the recipient's Activities in CiviCRM.

JoeMurray commented 9 years ago

Turns out, looking at shawnholt's instance, that Mandrill can successfully call back to more than one webhook per event. The data is coming back in to the extension and being put into some of the tables apparently correctly. Whether inserting some bogus records into these tables from webhook callbacks related to another CiviCRM instance is causing his problems and yours remains to be determined from our testing.

Regarding your responses: 1) Navigating to the page won't provide the information since that would not post the information to your site that Mandrill's call of the webhook posts. Could you try shawnholt's advice, and just after https://github.com/JMAConsulting/biz.jmaconsulting.mte/blob/master/CRM/Mte/Page/callback.php#L42 in your copy of the file at /sites/default/civicrm_extensions/biz.jmaconsulting.mte/CRM/Mte/Page/callback.php, add:

watchdog('MTE', '%print_r', array('%print_r' => print_r( $reponse, TRUE)));

Then send a transactional email to a bad email address to cause a hard bounce and a few hours later check your Drupal Reports for this entry in the Watchdog table and post the printout of the array here?

2) Bulk will also be causing webhooks to fire back to CiviCRM, but the extension is not currently set up to be able to handle them as we have not been inserting a unique code we need to track the email that generated the callback.

3) We will investigate why creating an Activity is not happening for clicks, opens, and bounces as it should (see https://github.com/JMAConsulting/biz.jmaconsulting.mte/blob/master/CRM/Mte/Page/callback.php#L190 ).

Skyping with shawnholt just now: He was definitely having the webhook callback returning, the array from Mandrill looked fine, it was being inserted into tables like civicrm_mandrill_activity (for outbound) and civicrm_mailing_event_bounce for hard bounces, but in the UI the Contact Summary Page did not show the relevant email being disabled, nor an Activity being created.

Pradeep is on vacation for a few weeks, but we'll see if we can move this ahead in his absence.

robbrandt commented 9 years ago

Thanks for 1), that's the detail I needed. I'm not a Drupal expert by any means. Is it "Recent log messages" where I look for the results?

JoeMurray commented 9 years ago

Yes

robbrandt commented 9 years ago

TYPE MTE DATE Tuesday, October 7, 2014 - 18:41 USER Anonymous (not verified) LOCATION http://crm.domain.com/civicrm/ajax/mte/callback?mandrillSecret=xxxxx REFERRER
MESSAGE Array ( [0] => Array ( [type] => blacklist [action] => add [reject] => Array ( [reason] => hard-bounce [detail] => smtp;550 5.1.1 The email account that you tried to reach does not exist. Please try double-checking the recipient's email address for typos or unnecessary spaces. Learn more at http://support.google.com/mail/bin/answer.py?answer=6596 i47si33759416yha.133 - gsmtp [last_event_at] => 2014-10-08 01:38:37 [email] => testbounce@domain.com [created_at] => 2014-10-08 01:38:37 [expires_at] => 2014-10-15 01:38:37 [expired] => [subaccount] => [sender] => ) [ts] => 1412732317 ) ) SEVERITY notice HOSTNAME 54.244.78.85 OPERATIONS

JoeMurray commented 9 years ago

From what we see, the CiviCRM_mandrill_id header isn't being sent alongwith the message which is the reason the message isn't recorded in CiviCRM. How was the non-transactional email sent? From the mandrillapp or from CiviCRM? Normally, we insert a custom header into transactional emails sent from CiviCRM so that on the call back we can match up the event with the sent email.

robbrandt commented 9 years ago

Yes, sending from Civi is the entire point. Specifically, I was at: civicrm/activity/email/add?action=add&reset=1&cid=1&selectedChild=activity&atype=3

and clicked on the Send Email button.

pradpnayak commented 9 years ago

Hi robbrandt

I need more info about the error that you facing in recording bounces

  1. Can i get the civicrm version? Was the civicrm fresh installed from tarball or Upgraded?
  2. MTE version (1.3 or 1.4)? Fresh installed or upgraded?
  3. Can I get sites/default/files/civicrm/ConfigLog/Civicrm.[hash].log file or the content of the file?
  4. Does the site record Open and Clicks?
robbrandt commented 9 years ago

I'd like to resurrect this issue. I am trying this again, in a new CiviCRM implementation. It is Civi v. 4.5.2, Drupal 7, latest version, and MTE 1.4, just downloaded and installed. I followed the readme to the letter. I have disabled IDS.

Bulk sending works, and open and click tracking are accurately reported in the mailing report. Bounces are not.

Doing an individual mail correctly sends the mail. There doesn't seem to be any record of opens, clicks, or bounces. When I send by Actions -> Send an Email, TWO activities are created, "Email", and "Mandrill Email Sent". If I send the email simply by clicking on an email address, only the "Email" activity is created. Neither, however, results in any tracking of any kind.

I reserved judgment on whether it was working until I saw the result in the Mandrill activity page, and even after I went to the webhook page and clicked the Test button, which confirmed that 13 tests were successfully sent.

robbrandt commented 9 years ago

I sent you my log files via email, did you get it?

JoeMurray commented 9 years ago

Hi robbrandt, Did you send the files to me or Pradeep? What was the subject, approximate date, or From email so we can search in our spam folders?

Joe Murray, PhD President, JMA Consulting joe.murray@jmaconsulting.biz skype JosephPMurray twitter JoeMurray 416.466.1281

On Tue, Nov 4, 2014 at 1:15 PM, robbrandt notifications@github.com wrote:

I sent you my log files via email, did you get it?

— Reply to this email directly or view it on GitHub https://github.com/JMAConsulting/biz.jmaconsulting.mte/issues/56#issuecomment-61687317 .

pradpnayak commented 9 years ago

@robbrandt: I got your mail. We installed D7 with Civicrm 4.5.2 and MTE 1.4 on our JMA instance. We did not turned off IDS. We could record Bounces, Clicks And Open. The Activity gets recorded for Opens and Clicks but not for bounces (The Mandrill Bounces activity gets recorded when Group to notify is selected in Mandril Smtp Settings - Outbound Mail)

I think we should change the code to create Mandrill Bounce activity for Bounce mandril event ( @JoeMurray , Need your feedback on this).

@robbrandt: Can you check the Mailing Bounce report to check if bounces are recorded by CiviCRM or Can you check Apache Error or Drupal logs related to biz.jmaconsulting.mte and post here for refrence?

shawnholt commented 9 years ago

Fyi. I have still not been able to resolve on 2 stacks either. Shawn. On Oct 31, 2014 5:23 PM, "robbrandt" notifications@github.com wrote:

I'd like to resurrect this issue. I am trying this again, in a new CiviCRM implementation. It is Civi v. 4.5.2, Drupal 7, latest version, and MTE 1.4, just downloaded and installed. I followed the readme to the letter. I have disabled IDS.

Bulk sending works, and open and click tracking are accurately reported in the mailing report. Bounces are not.

Doing an individual mail correctly sends the mail. There doesn't seem to be any record of opens, clicks, or bounces. When I send by Actions -> Send an Email, TWO activities are created, "Email", and "Mandrill Email Sent". If I send the email simply by clicking on an email address, only the "Email" activity is created. Neither, however, results in any tracking of any kind.

I reserved judgment on whether it was working until I saw the result in the Mandrill activity page, and even after I went to the webhook page and clicked the Test button, which confirmed that 13 tests were successfully sent.

— Reply to this email directly or view it on GitHub https://github.com/JMAConsulting/biz.jmaconsulting.mte/issues/56#issuecomment-61333650 .

robbrandt commented 9 years ago

Was working on logs, while doing so I remembered that when I tested the Mailchimp extension I had to grant anonymous access to the Mailchimp plugin in Drupal. I see nothing about that in your instructions. I would think Drupal would block anonymous access without setting something up. What do I need to grant to anonymous to make this work?

JoeMurray commented 9 years ago

We're going to provide Activities for Hard Bounce, Spam, and Soft Bounce callbacks, the latter only when there have been enough soft bounces time within the time window used by CiviMail to merit disabling sending to the email. This will allow admins to recognize when and why a user's email was disabled.

I think that there is no permission problem on the callback url, but Pradeep will have to confirm.

@shawnholt and @robbrandt, would you be willing to test a beta version of this for us if we provide it tomorrow?

robbrandt commented 9 years ago

Sure, but I am a little confused about what your intent is. Why an Activity? Or are you not referring to the Bulk Mail functions? Hard bounces are must important to us in Bulk Mails. Will the "On Hold" data be automatically set when a hard bounce happens?

JoeMurray commented 9 years ago

We are already processing hard bounces etc, and it is available through the CiviMail report (at least on our test instances), but we felt it would also be useful for there to be an Activity for easy reference by staff users who are less technically minded.

Would it be possible for you to provide ssh access to pradeep (dot) nayak (at) jmaconsulting (dot) biz to an instance of yours that is not working so he can do some diagnostics? Alternatively, could you be available any time before 10am Eastern in the next day or two for a Skype call to have you do the same under his direction?

robbrandt commented 9 years ago

If needed, yes, but I am pacific time zone (California) and would appreciate a later time.

JoeMurray commented 9 years ago

8am Pacific is 9:30pm in Mumbai http://www.timeanddate.com/worldclock/meetingtime.html?iso=20141106&p1=224&p2=44

Would that work for you?

robbrandt commented 9 years ago

Ah, India. That's a tough one. Normally I would say evening my time, but that won't work this week. I will be available at 7am tomorrow or the next day, provided he confirms beforehand.

JoeMurray commented 9 years ago

I can commit on his behalf as he is working at that time tomorrow (and Friday). Can you post your Skype handle or how he can find you on Skype here? Thx.

robbrandt commented 9 years ago

Thanks. I will email it to him.

robbrandt commented 9 years ago

After Skyping with Pradeep I can confirm that the extension is working as designed. The issue is of documentation/understanding. In version 1.4, bounces do not set the "On Hold" value like native bulk mailing does, nor does it tally bounces in the report of an individual bulk mailing in CiviMail. What it DOES do is show bounces in the Mail Bounce report (civicrm/report/instance/29?reset=1) as Mandrill Hard, Mandrill Soft, Mandrill Reject, etc.

If I understand Pradeep correctly, support for On Hold will be added in 1.5 (shortly) and bulk mail reports in 2.0. I am very happy about this, because all we want is to use Mandrill in CiviCRM just like any other mail server.

shawnholt commented 9 years ago

Thanks for that clarification - I was under the impression from my conversation with Joe a month or so ago, that it was designed to also set On Hold

It seems like the module does two primary tasks:

  1. Update the mailing report (to see how effective the mailing was)
  2. Mark an email as bounced (to avoid sending to bad email addresses). This is done by creating an activity for the contact and updating the On Hold, etc according to Civi bounce processing logic...

2 is extremely important and (I would argue) completely independent of #1.

I point this out because the bounce callback looks for the specific mailing identifier and if not found it aborts. I would argue that no matter what mail triggered the callback (transnational, bulk, etc.) comes from, if an email address is marked bad, that information should be processed (activity recorded, on-hold.)

Obviously mailing reports for bulk are important, but (IMO) contact bounce processing is at least as important.

I realize that 1.5 is almost out, but please consider adding bounce processing even if you can't identify the mailing id.

shawnholt commented 9 years ago

Also - Would be happy to test a beta

robbrandt commented 9 years ago

For us, bounces from bulk mail are far more important, and are more likely to contain addresses that are old.

robbrandt commented 9 years ago

Status update? Is there a beta on the horizon?

pradpnayak commented 9 years ago

https://github.com/JMAConsulting/biz.jmaconsulting.mte/archive/v1.5.zip

I have tagged this as a pre-release mte version 1.5

pradpnayak commented 9 years ago

@robbrandt and @shawnholt, would you do QA on this for us?

robbrandt commented 9 years ago

Yes, hopefully today or tomorrow.

shawnholt commented 9 years ago

One quick thing - I believe you include a webhook permission. if that is not enabled for anonymous it will not work - correct? If that is the case i suggest the default be to enable it - i don't know how it would get authenticated anyway (although it is probably useful to have permission.)

pradpnayak commented 9 years ago

No, you don't need to give any permission for anonymous user. The callback url is public and don't need any permission. Your issue was regarding improper bounce type name stored in civicrm_mailing_bounce_type this was due to upgrade of CiviCRM from 4.4.x to 4.5. The data type of name field has changed from enum to varchar. Please upgrade the code and run extension upgrade utility to fix the corrupt data. Sent from my BlackBerry 10 smartphone.

JoeMurray commented 9 years ago

This issue is more of a discussion thread at the moment. I'm going to close it and ask people to post new issues for anything they find in the new release. Thanks very much for your engagement! It helps make the software better. Cheers, Joe