search

3stack-software / celery-shoot

Celery client for Node.js
BSD 3-Clause "New" or "Revised" License
1 stars 1 forks source link

Usage example in Meteor / mysterious assertion error #2

Closed optilude closed 9 years ago

optilude commented 9 years ago

Hi,

I'm trying to use this in Meteor via 3stack:meteor-connect.

Celery is configured with:

app = Celery('jiraflow',
         broker='amqp://',
         backend='amqp://',
         include=['proj.tasks'])

Meteor is started with env CELERY_BROKER_URL="amqp://" meteor

Method is defined with:

runCeleryTask: function(left, right) {
    this.unblock();

    var task = Celery.createTask('proj.tasks.add');

    try {
        return Meteor.wrapAsync(task.invoke, task)([left, right]);
    } catch(e) {
        console.error(e);
    }
}

This is called from the client with: Meteor.apply("runCeleryTask", [3, 3], {wait: true}, function(err, result) { console.log("Done"); console.log(err); console.log(result); }).

This fails with an error in the logs:

W20150720-23:16:07.845(1)? (STDERR) AssertionError: false == true
W20150720-23:16:07.845(1)? (STDERR)     at Task.CeleryTask_sendMessage [as sendMessage] (/Users/maraspeli/.meteor/packages/3stack_celery-shoot/.4.0.1.1gklmv3++os+web.browser+web.cordova/npm/node_modules/celery-shoot/src/lib/Task.js:182:5)
W20150720-23:16:07.845(1)? (STDERR)     at Task.CeleryTask_invoke [as invoke] (/Users/maraspeli/.meteor/packages/3stack_celery-shoot/.4.0.1.1gklmv3++os+web.browser+web.cordova/npm/node_modules/celery-shoot/src/lib/Task.js:104:10)
W20150720-23:16:07.845(1)? (STDERR)     at packages/meteor/helpers.js:118:1
W20150720-23:16:07.845(1)? (STDERR)     at [object Object].Meteor.methods.runCeleryTask (server/query.import.js:41:55)
W20150720-23:16:07.845(1)? (STDERR)     at maybeAuditArgumentChecks (packages/ddp/livedata_server.js:1617:1)
W20150720-23:16:07.846(1)? (STDERR)     at packages/ddp/livedata_server.js:648:1
W20150720-23:16:07.846(1)? (STDERR)     at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1)
W20150720-23:16:07.846(1)? (STDERR)     at packages/ddp/livedata_server.js:647:1
W20150720-23:16:07.846(1)? (STDERR)     at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1)
W20150720-23:16:07.846(1)? (STDERR)     at [object Object]._.extend.protocol_handlers.method (packages/ddp/livedata_server.js:646:1)

I can't really understand what's going on :(

There doesn't seem to be docs explaining how to use it with Meteor, and 3stack:celery seems to be a different thing now? What am I missing?

Martin

nathan-muir commented 9 years ago

Hi @optilude,

  1. The AssertionError points to assert(self.client.connection.state === 'open'); - This means it's not connecting to AMQP correctly.
  2. It's probably because the URL parsing is different with the new AMQP library - Try the full URL amqp://guest:guest@localhost:5672//. (eg, must include, user, pass, host, port, vhost)

  3. You don't need to do Meteor.wrapAsync - running celery-connect / celery-shoot under meteor automatically mixes in task.invokeSync - which returns a Future. eg:

    runCeleryTask: function(left, right) {
 this.unblock();

 var task = Celery.createTask('proj.tasks.add');

 try {
   return task.invokeSync([left, right]).wait().result;
 } catch(e) {
   console.error(e);
 }
}
  4. Yeah, I haven't had the chance to write up some decent docs! I'll try get some meteor examples added.
optilude commented 9 years ago

Hi!

Those changes cause the following error to appear in the meteor output:

W20150721-00:05:38.519(1)? (STDERR) 
W20150721-00:05:38.520(1)? (STDERR) /Users/maraspeli/.meteor/packages/3stack_celery-shoot/.4.0.1.1gklmv3++os+web.browser+web.cordova/npm/node_modules/celery-shoot/src/lib/Task.js:245
W20150721-00:05:38.521(1)? (STDERR)                 status = message.status.toLowerCase();
W20150721-00:05:38.521(1)? (STDERR)                                         ^
W20150721-00:05:38.521(1)? (STDERR) TypeError: Cannot call method 'toLowerCase' of undefined
W20150721-00:05:38.521(1)? (STDERR)     at Consumer.CeleryTask_onMessage [as messageHandler] (/Users/maraspeli/.meteor/packages/3stack_celery-shoot/.4.0.1.1gklmv3++os+web.browser+web.cordova/npm/node_modules/celery-shoot/src/lib/Task.js:245:41)
W20150721-00:05:38.521(1)? (STDERR)     at Consumer._onContent (/Users/maraspeli/.meteor/packages/3stack_celery-shoot/.4.0.1.1gklmv3++os+web.browser+web.cordova/npm/node_modules/celery-shoot/node_modules/amqp-coffee/bin/src/lib/Consumer.js:341:21)
W20150721-00:05:38.521(1)? (STDERR)     at Consumer._onContent (/Users/maraspeli/.meteor/packages/3stack_celery-shoot/.4.0.1.1gklmv3++os+web.browser+web.cordova/npm/node_modules/celery-shoot/node_modules/amqp-coffee/bin/src/lib/Consumer.js:4:61)
W20150721-00:05:38.521(1)? (STDERR)     at Connection._onContent (/Users/maraspeli/.meteor/packages/3stack_celery-shoot/.4.0.1.1gklmv3++os+web.browser+web.cordova/npm/node_modules/celery-shoot/node_modules/amqp-coffee/bin/src/lib/Connection.js:580:24)
W20150721-00:05:38.521(1)? (STDERR)     at AMQPParser.<anonymous> (/Users/maraspeli/.meteor/packages/3stack_celery-shoot/.4.0.1.1gklmv3++os+web.browser+web.cordova/npm/node_modules/celery-shoot/node_modules/amqp-coffee/bin/src/lib/Connection.js:4:61)
W20150721-00:05:38.522(1)? (STDERR)     at AMQPParser.emit (events.js:98:17)
W20150721-00:05:38.522(1)? (STDERR)     at AMQPParser.parseContent (/Users/maraspeli/.meteor/packages/3stack_celery-shoot/.4.0.1.1gklmv3++os+web.browser+web.cordova/npm/node_modules/celery-shoot/node_modules/amqp-coffee/bin/src/lib/AMQPParser.js:144:19)
W20150721-00:05:38.522(1)? (STDERR)     at AMQPParser.frameEnd (/Users/maraspeli/.meteor/packages/3stack_celery-shoot/.4.0.1.1gklmv3++os+web.browser+web.cordova/npm/node_modules/celery-shoot/node_modules/amqp-coffee/bin/src/lib/AMQPParser.js:87:16)
W20150721-00:05:38.522(1)? (STDERR)     at AMQPParser.frame (/Users/maraspeli/.meteor/packages/3stack_celery-shoot/.4.0.1.1gklmv3++os+web.browser+web.cordova/npm/node_modules/celery-shoot/node_modules/amqp-coffee/bin/src/lib/AMQPParser.js:64:21)
W20150721-00:05:38.522(1)? (STDERR)     at AMQPParser.header (/Users/maraspeli/.meteor/packages/3stack_celery-shoot/.4.0.1.1gklmv3++os+web.browser+web.cordova/npm/node_modules/celery-shoot/node_modules/amqp-coffee/bin/src/lib/AMQPParser.js:51:21)
=> Exited with code: 8

However, I can see log output in the celery process that makes it seem like the task is being run.

No result makes it back to the client.

nathan-muir commented 9 years ago

Hmm, can you try setting these in your python celery config

CELERY_RESULT_SERIALIZER = 'json'

CELERY_TASK_SERIALIZER = 'json'
optilude commented 9 years ago

That worked!

So, documentation wise, I'd suggest:

$ meteor add 3stack:celery-connect

Celery config (basic adaption of the "next steps" example in the Celery docs):

proj/__init__.py: empty

proj/celery.py:

from __future__ import absolute_import

from celery import Celery

app = Celery('proj',
             broker='amqp://',
             backend='amqp://',
             include=['proj.tasks'])

# Optional configuration, see the application user guide.
app.conf.update(
    CELERY_TASK_SERIALIZER='json',
    CELERY_RESULT_SERIALIZER='json'
)

if __name__ == '__main__':
    app.start()

proj/tasks.py:

from __future__ import absolute_import
from jiraflow.celery import app

@app.task
def add(x, y):
    return x + y

@app.task
def mul(x, y):
    return x * y

@app.task
def xsum(numbers):
    return sum(numbers)

Start RabbitMQ:

$ rabbitmq-server

Start Celery (from the directory where proj lives):

$ celery -A proj worker -l info

Meteor method example:

Meteor.methods({

    runCeleryTask: function(left, right) {
        this.unblock();

        var task = Celery.createTask('jiraflow.tasks.add');

        try {
            return task.invokeSync([left, right]).wait().result;
        } catch(e) {
            console.error(e.stack);
        }
    }

});

Meteor Startup parameters:

$ env CELERY_BROKER_URL="amqp://guest:guest@localhost:5672//" meteor

Example calling the method:

Meteor.apply("runCeleryTask", [3, 3], {wait: true}, function(err, result) {
    console.log("Done");
    console.log(err);
    console.log(result);
})

Obviously kind of contrived, but it seems to work.

nathan-muir commented 9 years ago

Awesome.

I had a quite go at updating the docs @ https://github.com/3stack-software/meteor-celery-connect

But I'll happily add your outline as a Getting-Started.md

optilude commented 9 years ago

Docs look good!

Would maybe add to the 3stack:celery package a note that it is deprecated?

— Sent from Mailbox

On Tue, Jul 21, 2015 at 9:15 AM, Nathan Muir notifications@github.com wrote:

Awesome. I had a quite go at updating the docs @ https://github.com/3stack-software/meteor-celery-connect

But I'll happily add your outline as a Getting-Started.md

Reply to this email directly or view it on GitHub: https://github.com/3stack-software/celery-shoot/issues/2#issuecomment-123209031