Persistence in the BPDM orchestrator

[maximilianong]

Description

To enhance the robustness and scalability of the BPDM (Business Process Data Management) orchestrator by implementing improved exception handling and persistence mechanisms, especially in scenarios involving orchestrator unavailability and mass-upload operations.

Exception Handling Improvement:

As a user, I want the system to handle BPDM orchestrator unavailability or crashes gracefully, so that I can be informed about the failure and know to restart the process.

Persistent Storage Integration:

As a system administrator, I want the BPDM orchestrator to use persistent storage instead of in-memory storage, so that the system can handle larger volumes of data and maintain state across restarts.

• Transition the BPDM orchestrator from in-memory storage to a persistent storage solution (e.g., database).
• Ensure data integrity and consistency during the transition.
• Implement data backup and recovery mechanisms to protect against data loss.

Mass-Upload Handling:

As a user, I want the system to handle mass-upload operations efficiently, so that I can upload large datasets without hitting system limits or experiencing failures.

• Provide an option to enable persistence in the gate to handle uploads exceeding the in-memory storage limit.
• Implement load balancing and optimized data processing techniques to efficiently manage mass-upload operations.

Impact

By implementing these features, the BPDM orchestrator will become more resilient to failures, capable of handling larger datasets, and better equipped to manage high-volume data uploads.

Additional information

• [x] I'm willing to contribute to this feature / + @nicoprow

[stephanbcbauer]

After a brief consultation with Sebastian, the feature is set to 'backlog' status. Associated committer is also fine with it.

[maximilianong]

This is part of the 24.08 release.