Support for Azure Blob Storage for Archival

[dhiaayachi]

Is your feature request related to a problem? Please describe. Currently Archival is only supported for Gcloud and AWS S3, our org uses Azure as its cloud provider, hence we would like to use Azure Storage for Archival.

Describe the solution you'd like We would like a plugin based solution where we can go ahead and implement our own code and pass them to something like the Server Options or any native support from Temporal side is also welcome.

Describe alternatives you've considered We did try to go ahead and implement the Azure dependencies on our own but since it requires changes in the original source code, it would add a significant overhead on our team to maintain the codebase and keep it updated with newly released versions, we could not proceed with it.

[dhiaayachi]

Is your feature request related to a problem? Please describe.

Currently, Archival, which automatically backs up Workflow Event Histories and Visibility records, only supports Google Cloud and AWS S3. Our organization uses Azure as its cloud provider, so we need a way to use Azure Storage for Archival.

Describe the solution you'd like

We'd like to see a plugin-based solution for Archival, allowing us to implement our own code and integrate it with the Temporal Server. Alternatively, native support for Azure Storage in Temporal would be highly appreciated.

Describe alternatives you've considered

We attempted to implement the Azure dependencies ourselves, but due to the required changes in the original source code, it would create a significant overhead for our team to maintain and update the codebase. This approach was deemed unsustainable.

References

• Archival: https://docs.temporal.io/clusters#archival
• Temporal Server Configuration: https://docs.temporal.io/references/configuration
• Custom Archiver: https://docs.temporal.io/self-hosted-guide/archival#custom-archiver

We believe that adding Azure Storage support to Temporal Archival would greatly benefit users like us who rely on Azure for their cloud infrastructure. It would enhance the platform's versatility and appeal to a wider range of users.

[dhiaayachi]

Thank you for your feature request. We understand that you would like to use Azure storage for archival.

Currently, Temporal's archival feature supports only Google Cloud Storage (GCS), AWS S3, and the local file system. We are looking into adding support for other cloud providers in the future.

In the meantime, we suggest you try to use a custom archiver to support Azure storage. You can follow these steps to create a custom archiver:

1. Create a new package in /common/archiver.
2. Implement the HistoryArchiver and VisibilityArchiver interfaces.
3. Update the archiverProvider object in /common/archiver/provider/provider.go to know how to create an instance of your archiver.
4. Add configs for your archiver to the config/development.yaml file.

For more information, please refer to the Self-hosted Archival setup documentation.

We appreciate your feedback and will consider your request for native support for Azure storage in future releases.

[dhiaayachi]

Thank you for your feature request! We appreciate you taking the time to suggest this.

Currently, Temporal's Archival feature only supports AWS S3 and Google Cloud Storage. We understand your need for Azure Storage support. While we don't have native support for Azure Storage at this time, a plugin-based solution could potentially address this.

We encourage you to explore creating a custom Archiver for Azure Storage, following the instructions provided in our Self-hosted Archival Setup documentation.

This approach would involve implementing the necessary interfaces for the Azure Storage provider and integrating it with Temporal's Archival system.

While this would require development effort, it would allow you to use Azure Storage for Archival without waiting for native support.

Let us know if you have any further questions about implementing a custom Archiver. We're happy to help.

[dhiaayachi]

Thank you for the feature request. Currently, Temporal directly supports archival to GCloud and AWS S3 but does not directly support Azure.

As a workaround, we suggest you consider implementing a custom archiver to handle Azure Blob Storage. The documentation on Creating a Custom Archiver provides a comprehensive guide on how to achieve this.

While implementing your own solution may seem like an overhead, it offers flexibility and control over how you archive your workflow histories. Additionally, you can potentially contribute the custom archiver back to the Temporal community for others to benefit from it.

[dhiaayachi]

Thank you for your feature request. We appreciate you bringing this to our attention. Currently, we only support AWS S3 and Google Cloud Storage for archival.

While we are looking into supporting Azure Storage in the future, it's not yet available. As a workaround, you can consider implementing a custom archiver for Azure Storage, as described in the "How to create a custom Archiver" section of the "Self-hosted Archival setup" page (https://docs.temporal.io/self-hosted-guide/archival#custom-archiver).

This solution allows you to leverage Azure Storage for archival while providing flexibility and control over your specific implementation.

[dhiaayachi]

Thank you for the feature request. We are currently focused on improving our existing archival solutions for AWS S3 and Google Cloud. However, you can implement your own Azure archival solution by using the plugin based solution mentioned. You can refer to the Self-hosted Archival setup section for detailed information on creating a custom archiver.