Redis Background Tasks & Sockets

[thehydroimpulse]

I was thinking of having an option to control redis tasks. This could be embedded in the socket events @on and @emit.

When a background task finished, it would emit a socket event. You could then listen and emit jobs.

class App.VideosController extends App.ApplicationController
  @on 'uploaded::video', 'complete', type: 'job'

  complete: (job) ->
    # Alert User that his / her video is uploaded:
    # The completed job would send specific information.
    # In this example, the job would return:
    job.uploader.id
    job.uploader.username
    job.video.hash
    job.video.url
    job.video.encoding
    # You can then alert a flash message, etc... displaying this information.

You could extend this functionality quite a bit. The api @on could be split up into @jobOn if it gets a little confusing but that's super little.

Any thoughts?

[lancejpollard]

When a background task finished, it would emit a socket event. You could then listen and emit jobs.

I love this idea. Very cool, wanna hear more about that.

It seems this would be a very clean way to do file uploading. Also other common background tasks like emailing or fetching and processing a webpage.

Not sure how I feel about putting that @on code in the controller instead of the model though, will think about more. Socket events do seem like they should come from the controller from a high level, but maybe they're hooked into a callback defined on the model, something like:

class App.Video extends Tower.Model
  @after 'create', 'upload'

  upload: (done) ->
    @enqueue 'upload', @get('id')

  @upload: (id, done) ->
    # do upload...
    done() # which calls back to the socket `@on` handler in the controller.

What are your thoughts?

[lancejpollard]

Though I did hear people are starting to avoid callbacks like @after 'create' and choosing to define those actions in the controller, so maybe it is a preferred pattern putting it in the controller...

[thehydroimpulse]

I actually rather have that in the model tbh. As background jobs are a type of data source. Maybe a mix of both?

Instead of only having the controller have the control, maybe let the model have the hooks. So you would listen through the model?

Maybe we could implement some helper functions in the controller for jobs.

class App.VideoController extends App.ApplicationController
  App.Job.On 'upload', 'upload'

  upload: (job) ->
    # This could be a client-side controller that would then present
    # a message to the user in some way.

A mix of models and controllers would be perfect but we would need to find the perfect api..

Thoughts?

[thehydroimpulse]

But I also think that the controller should be responsible on acting on events. In this case it would easily allow a client-side controller to present another view, partial, or popup message. The controller could call a particular view class to act on the given data.

class App.ApplicationView extends Ember.View
  uploadComplete: (job) ->
    # Now you can update the current view to include some sort of alert (popup, partial, transition to a different view)
    # You can then access the correct data through `job`.

class App.VideoController extends App.ApplicationController
  App.Job.On "upload", "upload"

  upload: (job) ->
    # Call the view object
    # Or create a new flash message:
    @flash "success", "Your video has been uploaded. You can view it here -> #{job.video.url}"

In this particular example a specific user uploads a video. Because you don't want to send out this event to all connected users, this could be plugged into a constraints system. The job itself would pass a constraints based on the job type. In this case it would use an Constraint.Authentication or maybe even Constraint.Authorization to check if the user is allowed to view the model (Video).

This would be a very nice addition and would make developing these types of application extremely easy! You specify a few controllers, views, and jobs to process the data / upload files and you've got a working demo.

Thoughts?

[lancejpollard]

I like where you're going with this. This would definitely make creating a demo app very easy.

Seems like you know what should be done here, so I just have some high-level comments on the API.

First, when I ran into this situation (uploading videos, or even testing with resizing images but that's pretty fast), I did something like this:

# app/controllers/server/videosController.coffee
class App.VideosController extends App.ApplicationController
  create: ->
    video = App.Video.build(@params.video)

    # this is called after create, after the http response was sent back to the user,
    # so this just going to emit socket events
    video.on 'upload', (percentComplete) =>
      # emit only sends to current user, through web sockets.
      # the exact api wasn't/isn't implement, but basically just
      # send the current user who uploaded the video the data they need...
      json = video.toJSON()
      # maybe this is a computed property on the video or something...
      json.percentComplete = percentComplete
      @emit 'uploaded', json

    # normal save and responding to HTTP
    video.save (error) =>
      if error
        @render json: {error: error}, status: 404
      else
        @render json: json, status: 200

The way that would be implemented, all the "background job with redis" code is abstracted away into the Tower.Model base class, so you'd implement it like this:

class App.Video extends Tower.Model
  @after 'create', 'upload'

  upload: (done) ->
    @enqueue 'upload', @get('id')

  @upload: (job, done) ->
    job.on 'complete', done
    job # return job or something, or somehow attach this to the `video.on` event handler in the controller.

That way, you can run jobs from the command line without worrying about the controllers. But if a job was requested through the controller (uploading video), then you hook into it from the controller.

I don't like the way I did that in the controller though, and it should be done more like what you're saying (I like the @on controller method rather than calling App.Job.on, which is too meta :) So maybe something like this:

class App.VideoController extends App.ApplicationController
  # these 3 methods are equivalent
  @on 'upload', 'upload'
  @event 'upload'
  @hook 'upload'

  upload: (job) ->

Then whenever a hook is defined it will register a handler on a record when instantiated (abstracted away into some Tower controller mixin).

The only issue with doing that is it seems a little abstract and may be confusing to beginners. Some of these simple abstractions make it harder for people to get started, so just another factor to consider. That's why maybe just manually writing the handler with @emit like the first example could be preferred even though it's more verbose.

Either way (API-wise), there will be a clean way to send socket notifications to the user who just made the request, vs. everyone (or a subset of everyone). Socket.io uses broadcast vs. emit which I like, what do you think?

The last thing is the job object. Right now the tower-model/server/queue.coffee code encapsulates the logic for running background jobs using kue, and they have their own job object (and I like how they've built it). I wasn't really ever a fan of how background jobs were done in rails, as classes such as in resque:

class Archive
  @queue = :file_serve

  def self.perform(repo_id, branch = 'master')
    repo = Repository.find(repo_id)
    repo.create_archive(branch)
  end
end

Really, all jobs just have one method, perform, and emit events at different phases in their lifecycle. The kue api handles this better in my opinion. There's no need to create 1 class for each job type. Still, we need to think through how we want to represent/define jobs in Tower (haven't had much time for this yet).

What are your thoughts?