How to run functional tests?

[gforcada]

I guess it goes to @datakurre: we have some doctests that on an event handler queue a task but the task is not handled right away.

As it is a functional test a transaction.commit() is done automatically as soon as you save a browser form, but still the task is not processed.

Is there something else that one needs to add on the testing layer besides adding TASK_QUEUE_FIXTURE on the fixture bases?

[datakurre]

Bare TASK_QUEUE_FIXTURE only logs queued tasks for collective.taskqueue logger with info level. That's probably, because I didn't know the proper way for synchronous request. Now that would probably be plone.subrequest, but I would not like to change the existing test fixture (but yet another variant would be needed).

TASK_QUEUE_ZSERVER_FIXTURE does queue and execute the tasks as usual through ZPublisher queue – asynchronously. In practice, that would require polling some URL in Plone. Just sleeping would not be enough, because ZServer is configured to require socket traffic to loop its asyncore loops (where taskqueue server is registered). (Without socket traffic the asyncore loop is configured to poll workers only once about every 30 sec.)

[gforcada]

We noticed that TASK_QUEUE_ZSERVER_FIXTURE fixture.

By polling some URL, you mean, that between the action that queues and the action that checks if it's already done there should be at least one call to a random URL and that should be enough?

But then if the async task takes 1 second or 10 minutes is not something that the utility gives you any pointer about, right? i.e #6 would be needed.

[datakurre]

At first, as mentioned, adding something like "subrequest handler" as option next to "loggin handler" should make the tasks instant for tests... Maybe that should be considered.

By polling some URL, you mean, that between the action that queues and the action that checks if it's already done there should be at least one call to a random URL and that should be enough?

Both ...ZSERVER_FIXTURE and production setup use asyncore based dispatcher for tasks, making the execution depend on ZSERVER asyncore loop. And the loop "runs faster" when there's any http (or other zserver connected) traffic. In normal usage that's not a problem, because http requests triggering tasks also keep the loop in motion. In test setup the traffic may be so optimized that the task is not send to ZPublisher without a extra request. (The redis integration connects redis' sockets to asyncore socket map, which fixes this issue on redis.)

So, if just queuing task and commit in functional test is not enough to call the task view (it still may be), call to any plone url should be enough to ensure that the task is dispatched to zpublisher. Then about result...

But then if the async task takes 1 second or 10 minutes is not something that the utility gives you any pointer about, right? i.e #6 would be needed.

That's still missing, because all my use cases have had resulted to some change in ZODB, effectively making the statues readable from ZODB (object workflow state, some flag/timestamp like attribute, etc...).

Because every task has unique id and task requests have special marker interface, it should be doable to register zpublisher request lifetime event handlers (like IPubEvent s) and guess the state from response status code.

[gforcada]

@datakurre thanks for the long explanation, zope internals are definitely not my comfort zone :-)

[datakurre]

@gforcada Isn't it fascinating how Plone has always been running on top of asynchronous HTTP server, but still we have managed to make asynchronous jobs very hard :)

Gil Forcada Codinachs wrote:

@datakurre https://github.com/datakurre thanks for the long explanation, zope internals are definitely not my comfort zone :-)

— Reply to this email directly or view it on GitHub https://github.com/collective/collective.taskqueue/issues/19#issuecomment-183667474.

[gforcada]

I kept the issue open for no good reason 😓