dhiaayachi
commented
1 week ago Thanks for the feature request!
This is an interesting idea. However, Temporal's design is based on the concept of storing the entire history to provide reliable and consistent execution. Optimizing for space by removing history could lead to complexities in recovering state in case of failures.
You could achieve something similar by using a separate persistence layer for successfully completed workflows. You could store the final state and basic information in a separate database, while using Temporal for the core workflow execution.
Let me know if you'd like to explore this approach further!
Is your feature request related to a problem? Please describe.
Many of our workflows process a decent amount of activity data and conclude with a short final status. We like the retention window to preserve de-duplication but don't need the full history for successfully exited workflows. However, Temporal naturally stores the full history for the entire retention window, wasting space in Cassandra.
Describe the solution you'd like
One solution to the above dilemma would be offering the ability to configure the system (perhaps on a per-workflow definition or invocation basis) where the developer indicates that the full history is unnecessary for successful workflows. When enabled, the workflow ID and final state would remain available for de-duplication and final-state retrieval, but the rest of the history (or at least the larger parts, such as data payloads) could be optimized out of storage.
Per-SDK Tickets