no clear how to create multiple queues

[datakurre]

Agreed. I need to add more comprehensive example into README.

Queues are defined as named utilities and new queues can be created with reusing the existing factories:

<utility factory="collective.taskqueue.redisqueue.RedisTaskQueue"
         name="my-redis-queue" />

my-redis-queue should be passed as queue=my-redis-queue to taskqueue.add and used as queue my-redis-queue -argument for new consuming server (servers are always queue specific)

    <taskqueue-server>
      name ${:_buildout_section_name_}
      queue my-redis-queue
    </taskqueue-server>

name should be instance name, because it would be used to mark which instance is currently processing the task (from Redis queue)

[djay]

I mean, the obvious way to have two different queues on the same instance is

<taskqueue-server>
  name ${:_buildout_section_name_}
  queue queue1
</taskqueue-server>
<taskqueue-server>
  name ${:_buildout_section_name_}
  queue queue2
</taskqueue-server>

but that doesn't work.

[datakurre]

@djay It's not that automatic yet. Sorry :/ queue1 and queue2 won't currently exist without custom utility-directives as shown in my first comment. Yet, we can still tune this, of course :)

Currently, c.taskqueue ships with only two queues available by default: default as instance-local volatile task queue and redis as distributed Redis task queue. More queues would require more utilities to be registered.

What is the use case for many queues? I can only come up with non-equal balancing / priorization of tasks.

To summarize, how configuration works:

1. Queues are global named utilities. c.taskqueue ships with default and local, but more can be registered in add-ons.
2. <product-configuration/> is required only to define default-queue for taskqueue.add-method and connection parameters for Redis-queue.
3. <taskqueue-server/> is required to consume a queue.

Technically, it would be possible to use to add <taskqueue-queue/>-type (used in zope-conf-additional) to dynamically register queues on start-up, but it would be a little bit zope.conf misuse, because e.g. plone.app.debugtoolbar would show those queues as servers.

[datakurre]

@djay

My current reasoning here is that because you need to always specify queue name for non-default queues for taskqueue.add, it would make sense to used fixed queues names in add-on and therefore also configure queues with zcml in add-on. Then it would be required to have only Redis-configuration in <product-configuration /> and <taskqueue-server/> on consuming instances.

<taskqueue-server/> cannot be enough for creating queues, because those would be only on consuming instances.

Enhancement ideas are still welcome for this :)

[djay]

Ah that explains the issue I'm having with with custom queue names. It's not at all obvious that queues names are fixed without code. Different queues is very important for example I might have one for serialised requests another for parrellel. I might have one hooked up to sales force, another to zope. One for doing email on one server and another for video processing on another dedicated server. It would be really good to support this some how On 14 Nov 2013 04:25, "Asko Soukka" notifications@github.com wrote:

@djay https://github.com/djay

My current reasoning here is that because you need to always specify queue name for non-default queues for taskqueue.add, it would make sense to used fixed queues names in add-on and therefore also configure queues with zcml in add-on. Then it would be required to have only Redis-configuration in and on consuming instances.

cannot be enough for creating queues, because those would be only on consuming instances.

Enhancement ideas are still welcome for this :)

— Reply to this email directly or view it on GitHubhttps://github.com/collective/collective.taskqueue/issues/4#issuecomment-28414722 .

[djay]

Doesn't seem so friendly to require two different kinds of configuration.

What about something like this

<taskqueue>
  queue writes
  type redis
  redis_unix_socket_path ${buildout:directory}/var/redis.sock
</taskqueue>
<taskqueue>
  queue email
  type local
</taskqueue>

<taskqueue-server>
  name ${:_buildout_section_name_}
  queue writes
</taskqueue-server>

[datakurre]

@djay syntax should be possible; I just need learn little more about Zope components...

[datakurre]

@djay

Ok. Can do that. Changes:

• support will be completely removed
• no queues will be defined by default
• taskqueue.add will default to alphabetically first queue and give a warning if that's different from the default

ZConfig will see task queues as ZServer.servers (developer will see that only in plone.app.debugtoolbar. I don't know yet, how to add custom types into Zope.

[djay]

Maybe having a default local queue defined if no others are defined is a good idea. At the moment the if your queue isn't found, it will send to default and that doesn't exist so it goes no where.

[djay]

or maybe unless you override the "default" queue, default is a local queue. That way if you queue a task you are guaranteed it will always run, even if you got the queue=name param wrong. At the moment it just seems to go no where.

[datakurre]

@djay

Currently, taskqueue.add gives a warning, when given queue is not found (and tells where the task is send). If no queues are configured, it raises an AssertionError.

Without <product-configurations /> there's no way to change the default queue. That's why I dropped it for now. Also, it may not be good that task can be send to volatile queue, which has no consumer.

Queues are still utilities and currently "unnamed" utility is not used. That may allow some default behavior later (and then it could be overridden with zcml-overrides.

Having a default consumer (<taskqueue-server />) is not possible, because that line is required to properly start the consumer.

[djay]

cool.

However its perhaps a little confusing that for the local queue, HAS to be on the same instance. But I guess you can just throw an exception if someone defines a local queue without a corresponding on the same instance.