[celery] Investigate how to deal with several tasks chaining on multiple queues in cube generation

[raphaelrpl]

Definition

The following tasks describes how to build a data cube:

• blend - Apply temporal composition on Irregular Data Cube
• warp - Re-project spatially scene band
• merge - Merge all scenes in same day with the respective band
• publish - TODO
• upload - Deploy cube to AWS and INPE DPI Catalog

Problem

In order to parallel cube generation using celery tasks, we flask view to dispatch cube generation. First of all, we search on STAC Service to get available collections.

The following pseudo-code describes how to dispatch cube generation:

for each band:
    for each blend:
        for each merge:
            warps = map(lambda asset: warp(asset)))
            merges.append(warps)
        blends.append(merges)
# wait all blends and then publish data cube
chain(group(blends), publish.s()).apply_async()

However, once we defined chain and group of tasks to deal with cube generation, we are facing wrong order execution on celery worker listening individual queues. It seems related with Celery Bug.

Links

• Chains do not work with nested groups - 1671
• Celery Groups - out of orders
• Google Groups Discussion - Celery group and chain

[raphaelrpl]

Right now, the cube generation is using default celery queue (celery)