App creates 4 objects/emails per submission (move nugget processing into background worker)

[jahio]

The application is now generating 4 objects (nuggets) and corresponding e-mails per submission instead of the desired one.

I've done some research on why this is and found the cause: the application is just dog slow. The way it works is like this:

E-mail -> Postmark -> application

When an email is sent, it gets forwarded to Postmark, which does some work with it and sends an HTTP POST request to the application. However, when the application receives the email, it then has to do many time consuming tasks like read metadata from the image, potentially resize it, store it on Amazon S3 and then send out an e-mail. That gets worse for every photo submitted.

The problem comes from the fact that Heroku's router has a cap of 30 seconds per request. So from Postmark's perspective, it's sending the right information to the app, but it doesn't get a response from the app and after 30 seconds, Heroku, not the app, cuts off the request. Now Postmark is left with an error code saying the request "timed out", so Postmark re-queues the message for retry. Then it retries N number of times until it finally gets a positive response from the application - in other words, until the app takes less than 30 seconds to finish processing the submission. It just so happens that the number of total attempts here wound up being 4. It could just as easily be 2, or 7, or 25 (or whatever Postmark's cap is).

The solution: we need to move ALL nugget submission logic into a background process. The basic idea is to take the nugget submission from Postmark and hand it off to a background worker. This way, from Postmark's perspective, it will see a fast and correct response and not re-try the submission. Then, the background worker takes whatever time it needs to process the submission. Meanwhile, more submissions may come in that go to Postmark, and therefore to the app, none of which get hung or tripped up because the application is just accepting data - not processing it (yet).

This will necessitate a few Heroku addons and some additional cost (what that is has yet to be determined though I don't expect it to be all that much). I also anticipate that this will be a fairly challenging task that may take a little while.

[jahio]

This is nearly complete. I have a working implementation in production on Heroku right now. It just needs additional testing by the team. My testing shows that at least the main use cases are working as intended. Postmark ships up its request payload and gets a response back MUCH quicker now since all the app is doing is shoving this into a database and having a background worker dig it up and process it later.

[jahio]

Additionally, logged in users can get a look at the Resque-web interface at /resque. Auth goes through Devise.

[jahio]

Oh, and this did require creating another heroku "dyno" process to run Resque which jacked up your monthly bill to $70.50 according to the Heroku dashboard right now. It's possible that may need to go a little higher to facilitate a more robust redis instance and/or database in the future.

[chrisivester]

After further testing of the Prospector App we have found that the duplication of received emails has ceased. Also, the app seems to be running a little bit quicker as well. Thanks!

[jahio]

Glad to hear it Chris :) Going to mark this as closed.