AMQP for working queues between SCAN, INDEX, NLP

bamthomas commented 5 months ago

Is your feature request related to a problem? Please describe.

We have task queue that is either:

The parameter is batchQueueType

Then we have queueType that are used to instantiate queues used between pipeline tasks (scan, index, deduplicate...).

Describe the solution you'd like

We could imagine:

to have a single queueType parameter that is used by both
to consider each extract of a file/document as a task and use task queues, avoiding these two mecanisms

github-actions[bot] commented 4 months ago

This issue is stale because it has been open for 40 days with no activity.

github-actions[bot] commented 3 months ago

This issue was closed because it has been inactive for 20 days since being marked as stale.

github-actions[bot] commented 3 months ago

This issue was closed because it has been inactive for 20 days since being marked as stale.

ICIJ / datashare