search

proexchange / com.pesc.sparkpost

Integrates SparkPost to CiviCRM, so email can be sent out over the SparkPost service and bounces can be processed in CiviCRM
Other
9 stars 5 forks source link

Extension not appearing in Joomla #17

Closed tabletguy closed 8 years ago

tabletguy commented 8 years ago

For Joomla 3.5.1, under "CiviCRM » Administer CiviCRM", there is no entry for "SparkPost / CiviCRM Integration" on the "Add New" tab. I've refreshed the values, logged out and logged in again, but it's not there.

I would prefer not to download and install it manually.

Screenshots show installed and what's available in case that helps.

civicrm_existing_extensions_joomla_351

civicrm_new_extensions_joomla_351_a

civicrm_new_extensions_joomla_351_b

civicrm_new_extensions_joomla_351_c

tabletguy commented 8 years ago

I've posted a request on StackExchange. I can't check that this extension is released for Joomla, and that it's correctly showing up in the Joomla folder of releases, so hoping someone has an answer.

http://civicrm.stackexchange.com/questions/11164/not-all-extensions-appear-in-add-new-list

joseltorres commented 8 years ago

We're looking into what we need to do to get listed within civicrm.

tabletguy commented 8 years ago

Thanks. Does that mean it's not actually listed yet for Joomla? If so, must I install it manually? (I do want to get it up and working this week).

joseltorres commented 8 years ago

We wrote this extension to work with any CMS. While we have only tested it on Drupal and WordPress, I see no reason why it wouldn't work with Joomla. You will need to install it manually.

From:https://wiki.civicrm.org/confluence/display/CRMDOC/Extensions#Extensions-Installinganewextension

Manual installation of native extensions If you want to install a native CiviCRM extension which is not listed under Manage Extensions automatically:

  • Download the extension package from the download URL provided by the developer
  • Unzip / untar the package and place it in your configured extensions directory
  • If the package has been constructed properly - when you reload the Manage Extensions page the new extension should be listed with an Install link.

Please let us know how it goes.

MegaphoneJon commented 8 years ago

I'm on the extension review team. Please email info@civicrm.org with your civicrm.org user ID.

joseltorres commented 8 years ago

just sent the email, thanks.

tabletguy commented 8 years ago

I didn't realize that extensions could be listed as "CMS Independent" but still possibly not actually released/reviewed for every platform. (Not blaming anyone, but it seems an oversight in the process at some step).

I had thought other Joomla users were already using this extension.

Looking in the downloaded ZIP file, I don't think it's set up for manual installation. Based only on the manual installation instructions you posted here, I would have expected the zip file to contain a top level folder as the extension name. Thus, top level folder would be "comp.pesc.sparkpost" if I'm correct.

Is that what it's supposed to install as? Joomla default installation directory for extensions is: "/media/civicrm/custom_ext/"

Examples that I have already are:

So, before I unzip the files into the custom_ext directory, please confirm that your extension top level folder name should be com.pesc.sparkpost or let me know what it is otherwise?

And then I'll go ahead and install it manually.

Thanks in advance. Stephan

tabletguy commented 8 years ago

I might be reading the zip file incorrectly as well (on my local machine). Should the top level folder name be "com.pesc.sparkpost-1.3.1"?

If that's the case, then I was wrong when I said the top folder within the zip was missing. Sorry about that.

The program I used to open the zip started within that folder when displaying items.

mark-rodgers commented 8 years ago

The extension does not show on any platform under the "Add New" tab until reviewed, even Drupal and Wordpress users have to install the extensions manually as of this second.

When you extract it into your custom extensions directory the top level directory name should be com.pesc.sparkpost. Also we just released version 1.3.3. Check the Releases tab on GitHub for the latest version.

tabletguy commented 8 years ago

Thanks! I notice that some extensions have a separate "Manual Installation" section in their documentation, if you decide to add that. Thanks for verifying a top level directory name. I assume that files such as "sparkpost.php" should be directly under that top level name.

CiviCRM doesn't list the 1.3.3 release yet. I assume it's at https://github.com/proexchange/com.pesc.sparkpost/archive/1.3.3.zip

tabletguy commented 8 years ago

Have uploaded and then clicked "Refresh" on the Add Extensions page (it doesn't show up until clicking Refresh). I installed it. Need to complete installation steps (set up automatic cron, etc...)

mark-rodgers commented 8 years ago

Correct. Sorry about that, I briefly removed the 1.3.3 release after posting that comment due to me forgetting to update the version in the info.xml file. It should be back under releases.

Getting crons running in CiviCRM is relatively simple depending on how your system is setup. I would recommend visiting the CiviCRM docs for such information.

tabletguy commented 8 years ago

Installation and testing notes a) It's easy to make a mistake and put in the api_key value incorrectly because you have a - required" type of string (a string with spaces in it). I was stupid enough to double click to select and then paste, not realizing you had a few more words to replace. Then, when manually deleting the extra part, I don't know if there's supposed to be a line feed or not? That's how it shows up when I look at both the parameters in edit more AND in the log file.

My job is returning an error message. Since this is a first (?) Joomla install, I'm posting it here to see if I did everything correctly or not. Note that it says "Finished execution of SparkPost Fetch Bounces with result: Failure, Error message: Unauthorized."

The believe the API key is correct, but I'm not sure if I correctly removed all of the "extra" notes you had in the installed version, since you had some of it within angle brackets and some text outside of the angle brackets.

joomla_sparkpost_log_entry

mark-rodgers commented 8 years ago

If you are getting the error Unauthorized that means SparkPost is not accepting the API key you supplied. It looks like you are missing the line break after the api key, which is going to pull "events=etc,etc,etc" into your API key value, likely the reason it is saying Unauthorized. Make sure each parameter is on a new line. Also remove " - optional" from the end of the events and date_filter lines.

Your scheduled job should be as follows: image

tabletguy commented 8 years ago

Great! I had removed the line feed accidentally because I was trying to clean up.

Now getting "Finished execution of SparkPost Fetch Bounces with result: Success (a:0:{})"

Also, I see that I had to remove the two instances of " - optional" as well. Perhaps you could put ALL replaceable text within angle brackets, although the above example screen shot was very helpful. Some people might not notice that the " - optional" needed to be removed.

I won't know until I actually get a bounce if it's working, unless there's some way to test this?

One last question. Is this extension ONLY to get back mailing results (clicks, bounces, unsubscribe, etc), or is it needed to actually SEND the emails as well? I ask because even before installing it, my single test emails were going out, although I haven't run a group test yet.

Again, thanks in advance.

mark-rodgers commented 8 years ago

To test it you can create a new contact in CiviCRM and make up an long email account that you know will bounce @gmail.com, then add them to a mailing list and send out a bulk mailing to that group. SparkPost takes about 1-2 minutes to update their API results then you can run the scheduled job manually and check the mailing report to see if the bounce was tracked.

Just threw up a Joomla site, installed CiviCRM, installed SparkPost extension, ran through setup procedure then sent out a bulk mailing. Everything worked as intended.

confirmed working in Joomla: image

EDIT: In response to your last question, this extension does slightly alter mailings before they get send out by adding custom headers. You can send out some test mailings before you send out any real mailings if you would like to test to make sure it works.

Click-throughs and unsubscribes work without this extension. The biggest purpose of this extension is to get bounce processing reporting back from SparkPost to CiviCRM.

tabletguy commented 8 years ago

Thanks. In fact, that's how I already test. I have a group of email testors (5) plus 1 invalid email address. Sparkpost is reporting all 6 emails as accepted. I'll wait a few minutes and check again.

mark-rodgers commented 8 years ago

SparkPost should report it as bounced after a good minute or 2. If you go in now and manually run the SparkPost scheduled job what does the job log say?

tabletguy commented 8 years ago

Full message: Finished execution of SparkPost Fetch Bounces with result: Success (a:0:{})

I'm sending a second group, but I've added an invalid gmail address to this one. First group only had invalid addresses to own server, so not sure if those are being accepted anyway and then discarded (it's been 3 years since I set that part up).

So, give it another 10 minutes and I'll report again.

mark-rodgers commented 8 years ago

Success (a:0:{}) means no bounces were found. If SparkPost tracks any bounces the scheduled job will pull them from SparkPost and log them in JSON inside of the job log result, then mark them as bounces in the CiviCRM database.

Success (a:0:{ bounces will be listed here })

tabletguy commented 8 years ago

I'm not getting any bounces. It's reporting all delivered successfully. (Still getting result: Success (a:0:{})) Including obviously bad emails to:

Maybe it will catch up after some time. I know there's various time delays in email delivery, although the gmail one should have failed, I think.

tabletguy commented 8 years ago

Update: I tried sending from my personal account to those two addresses. The gmail address did bounce, but it took about 30 seconds before reporting back with a reject message. It's been a few minutes and I haven't gotten a bounce from the one sent to ahymsin.org yet.

joseltorres commented 8 years ago

Lets try editing your scheduled job to match this exactly (fill in your api key):

api_key=insertkeyhere
date_filter=0

Your issue may be caused by the date_filter=1. We do this by default to optimize the schedule job to run faster. The way this works is it filters the SparkPost API request to only return bounces from the last time the 'SparkPost Fetch Bounces' scheduled job successfully ran. So I can imagine a scenario where a scheduled job could run “successfully” after the bounce occurred in SparkPost but was not was tracked in CiviCRM correctly because a misconfigured scheduled job. The next time a scheduled job ran, even if now configured correctly, would miss this bounce because it would be before the window of time for bounces returned in the API call. If you disable the date filter in the API call with the date_filter parameter, it will default to getting the last 1000 bounce events, which should back-fill any missed bounces. So try the above params and run the 'SparkPost Fetch Bounces' scheduled job. Let us know how it goes.

tabletguy commented 8 years ago

Changed it to zero (too bad you can't put in a 1 or 2 hour overlap) but still no bounces. Getting: Finished execution of SparkPost Fetch Bounces with result: Success (a:0:{})

Did you want me to remove all the other items? After changing JUST the date_filter I have: Parameters raw (from db settings): api_key=4a<....>4 events=bounce,delay,policy_rejection,out_of_band,spam_complaint date_filter=0

copied exact, with the above two separate line breaks.

Sparkpost itself doesn't report any bounces either on the reporting pages. I could give someone reporting access to the account if needed.

tabletguy commented 8 years ago

I'm going to bed, back tomorrow. Will check Sparkpost again then.

joseltorres commented 8 years ago

give me a call tomorrow, see pesc.com, contact us page - call main line ask for jose 9am-5pm US Pacific time... we'll get this figured out.

tabletguy commented 8 years ago

Ok, will do. I'm in Minnesota, but just returned from overseas, so still jet lagged and tired.

tabletguy commented 8 years ago

Logged into Sparkpost this morning, and it's reporting 1 bounce (presumably the gmail one) Reporting is much more 'primitive' than Mandrill, because they don't show individual email addresses anywhere that I can find on the Sparkpost site.

CiviCRM email report does not show any bounces. Looking at the scheduled job log, there were no bounces reported there either, all the way back to yesterday.

joseltorres commented 8 years ago

@tabletguy thanks for connecting with me so we can figure this out. Noting for reference here: there was a bug in the code that made it so the params in the API call to sparkpost were incorrect and thus returned no bounce event if there were 3 or more "FROM" addresses in the civicrm settings. It has been fixed and a new release (1.3.4) has been created.