History Queue Deletion & Life Cycle

[dhiaayachi]

Is your feature request related to a problem? Please describe.

Our history queue logic supports creating new history queue by simply define a new category and task execution logics. However, there's no good story for deleting or stop using an existing queue due to the fact that there might always be some backlogs in the database.

Today, simply skipping task category registration on service start up will result in an undefined behavior:

• In 1.21, the server will panic if shard info sees a task categoryID but can't find the corresponding task category definition. (This is actually preferred now since it won't result in task loss and will be more explicit signal on mis-configuration)
• In master branch, server won't panic but queue won't start and the data will be left in shard info metadata.

We should have a proper design for history queue deprecation. This could be useful for disabling archival queue and in the future if more history queues needs to be dynamically created and deleted.

Describe the solution you'd like

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

[dhiaayachi]

Thank you for your feature request. It's a great idea to have a defined process for deprecating history queues. This is something we're actively thinking about, and your input is greatly appreciated!

In the meantime, as a workaround, you can consider using a naming convention for your history queues that makes them easier to identify and manage. This could involve using prefixes or suffixes to denote active and deprecated queues. For example, you could add a deprecated- prefix to the name of a queue you wish to stop using.

We are always looking for ways to improve Temporal. Your feedback helps us prioritize and develop new features.

[dhiaayachi]

Thank you for raising this issue. It's great to have a well-defined proposal for history queue deprecation.

We recognize the need for a robust solution to handle the removal or disabling of existing history queues, especially when dealing with potential backlogs.

Currently, Temporal doesn't explicitly support deleting history queues. You are correct, there's no easy way to "stop using" a queue because of possible backlogs.

For now, a workaround would be to ensure all remaining tasks in the queue are processed before removing the category definition. This would require careful coordination and potentially involves manual intervention to ensure all tasks are processed.

We are actively exploring options for a proper deprecation mechanism. We are looking at approaches that would allow for a controlled transition away from a history queue while preserving the data.

This is a high-priority feature request, and we will continue to update our documentation with the latest information and development plans as they become available.

[dhiaayachi]

Thank you for raising this important issue. We understand the need for a proper design for history queue deprecation.

Currently, there isn't a built-in mechanism for directly deleting or stopping an existing history queue. This is due to the potential for existing backlogs in the database.

However, you can achieve a similar effect by skipping task category registration during service startup. This approach will prevent the server from starting the queue, effectively disabling it. While this won't delete the existing data in the shard info metadata, it will prevent further processing of tasks related to that queue.

We are exploring options for a more robust solution for history queue deprecation in future releases.

For further information, please refer to our documentation on History Queue.