Not clear how to run jobs on the server side

[culebron]

I took the playground app and studied its source code. Then I tried to reproduce its functions but with the process function on the server side. And I'm missing something, because the methods aren't executed.

I'm not sure what's missing here, because after reading the docs and API it's still not clear. (I'd like to avoid setting up a standalone worker, because at this point it will be easier to get an earlier revision, where I had a usual collection with jobs, and just add rerunning/progress features.)

Is there a working example for this case?

Or should I just make a set of Meteor methods on my own, and call pause/restart/rerun/etc. from them?

lib/db.coffee

@ParsingJobs = JobCollection('parsing', {
    transform: (d) ->
        try
            res = new Job(ParsingJobs, d)
        catch e
            res = d
        return res
})

if Meteor.isServer
    Meteor.startup(->
        ParsingJobs.allow({
            admin: (user_id, method, params) ->
                # commented temporarily Roles.userIsInRole(Meteor.user(), ['admin'])
                true
        })

        ParsingJobs.startJobServer()

server.coffee

que = ParsingJobs.processJobs('parsing', {workTimeout: 10000}, (job, cb) ->
    # do some processing
    job.done('success')
    cb()

ParsingJobs.find({type: 'parsing', status: 'ready'}).observe
    added: ->
        que.trigger()

On the client I can just run a shell command:

x = ParsingJobs.find().fetch()[0]
x.rerun()

Result:

job_class.js:16 Uncaught Error: Job remote method call error, no valid invocation method found.

[vsivsi]

Hi, the playground app has a built-in server-side worker for 'cleanup' jobs. The code is here:

https://github.com/vsivsi/meteor-job-collection-playground/blob/master/play.coffee#L453-L474

What I don't see in any of your code above is the actual creation and saving of a 'parsing' job to be run. That is, the equivalent of these lines from the server-side of the playground app: https://github.com/vsivsi/meteor-job-collection-playground/blob/master/play.coffee#L453-L455

[vsivsi]

BTW, the error you received indicates that the client doesn't have a properly configured DDP connection to the server. That is, it does not appear to be a Meteor client instance. But that is a separate issue from the question you have asked about server-side workers.

[vsivsi]

If the "client" is a plain node.js program, then you need to manage the DDP connection yourself. See: https://github.com/vsivsi/meteor-job#intro https://github.com/vsivsi/meteor-job-collection-playground-worker/blob/master/work.coffee

[culebron]

Hi, Thanks, I'll check out the DDP setup.

The client in my case is just the browser. I did create some jobs in Meteor shell (system command line, $ meteor shell) this way:

> (new Job(ParsingJobs, 'parsing', {some data: 'numbers, text, etc.'})).save()
> (new Job(ParsingJobs, 'parsing', {some data: 'other numbers'})).save()

I looked at the lines you showed, and checked what I might have missed there. I thought it might have been these lines

https://github.com/vsivsi/meteor-job-collection-playground/blob/master/play.coffee#L378-L451

Here's my simple adaptation to get it running:

ParsingJobs.startJobServer()

Meteor.publish 'allJobs', (clientUserId) ->
    # This prevents a race condition on the client between Meteor.userId() and subscriptions to this publish
    # See: https://stackoverflow.com/questions/24445404/how-to-prevent-a-client-reactive-race-between-meteor-userid-and-a-subscription/24460877#24460877
    if @userId is clientUserId
        suffix = if @userId then "_#{@userId.substr(0,5)}" else ""
        return ParsingJobs.find({type: 'parsing'})
    else
        return []

ParsingJobs.events.on 'error', (msg) ->
    console.warn "#{new Date()}, #{msg.userId}, #{msg.method}, #{msg.error}\n"

# Only allow job owners to manage or rerun jobs
console.log('allowing')
ParsingJobs.allow
    manager: (userId, method, params) ->
        true

    jobRerun: (userId, method, params) ->
        true

    jobSave: (userId, method, params) ->
        true

    getWork: (userId, method, params) ->
        true

    worker: (userId, method, params) ->
        true

que = ParsingJobs.processJobs('parsing', {workTimeout: 10000}, (job, cb) ->
           # same as in the OP

But this didn't change anything.

Actually, does user interaction in the playground app change anything in cleanup jobs?

[culebron]

I checked out the DDP connection example. Not sure if it's what is really needed.

If I understand correctly, it is supposed to work like this:

[ standalone nodejs app: processJobs ] <= takes jobs via DDP connection <= [ Meteor server ]

The scheme that I'm trying to make work is this:

[ browser [ button to make a job rerun ] ] => [ Meteor server: processjobs that executes the job ]

I don't understand 2 things:

1. if the browser can make a DDP connection
2. if this connection is suitable to implement the latter scheme that I drew above

[vsivsi]

I'm assuming from the closed issue that you were able to resolve this...

[culebron]

@vsivsi no, I just left it stay. I still don't understand what was missing, and wrote a simple own implementation without jobs-collection (without being able to call Job.<method> , it is just a collection). I control the jobs via Meteor methods and have a similar watcher.

I'd appreciate a clearer description how to enable calls of Job.<method> from the browser. (to clarify, I tested this in the meteor-admin pages, being logged in.)

[vsivsi]

If your browser code is running in the Meteor environment, then this should be extremely simple.

If you create a repo with a minimal reproduction of the error you are getting (a Meteor app that runs except for the error), I'd be happy to take a look. The playground app is actually a good starting point. So, I'm not sure where you went wrong. This sounds like one of those bugs caused by a stray line of code that was accidentally deleted, but is easy to miss that it's gone.

[vsivsi]

Answers to your questions above:

if the browser can make a DDP connection...

The browser side of Meteor apps always have a DDP connection to the Meteor server they load from. job-collection defaults to using that connection. It is possible to connect to other Meteor servers as well (such as a specialized job server), but I don't think that's what you are trying to do here.

if this connection is suitable to implement the latter scheme that I drew above

Yes, absolutely. This is precisely how the playground app works (although it doesn't control jobs on the server, it easily could simply by moving the worker code for that collection there).

[culebron]

Ok, sounds reassuring, will do!