Object Storage - asynchronous replication

Izaia64 commented 2 years ago

As a user I want to be able to replicate object storage between regions I want to be able to consume these features through standard APIs / OVHcloud Customer panel Replication between 2 buckets, asynchronous replication (aprox. 15 minutes)

We will enable automatic and asynchronous replication of objects contained inside a source bucket to one or more destination buckets. Those destination buckets can be in one single region or dispatched throughout several regions according to your needs.

Async replication will allow customers to:

make exact copies of objects (all metadata will also be replicated)
address synchronization of data between several teams use cases
transition copies of your data into other storage classes to optimize costs
augment high resiliency by having your data replicated across multiple regions

What will be provided is a S3 compatible REST API to put, delete, download a replication configuration file that allows for definition of :

destination buckets
filters to select which objects to replicate
rule priorities

The following configuration elements will be supported:

replication to multiple destination buckets (including delete operation)
replication to buckets located across multiple regions

The following configuration will NOT be supported for now:

replication to buckets owned by a different account (destination buckets are not owned by the same account that owns the source bucket)
transition copies to COLD STORAGE (deep archive) class
transition to other storage classes
retrieval of replication metrics (Nb of objects pending replication, Bytes pending replication, Operations failed replication, Replication latency (delta between buckets), Thoughput)
replication of existing objects i.e objects that were present before replication was applied

Izaia64 commented 2 years ago

This feature is in development and targeted for Q2 2023 (April/May)

loliee commented 1 year ago

This feature will not be compliant with SSE-C encrypted objects right ? It will be compliant with SSE-S3 encrypted objects ?

Thanks

lason-ovh commented 1 year ago

This feature will not be compliant with SSE-C encrypted objects right ? It will be compliant with SSE-S3 encrypted objects ?

Thanks

Hi, this feature will not be compliant with SSE-C nor SSE-S3 for now:

for SSE-C, additional work will be needed so in the first iteration, it will not be supported yet
SSE-S3 (and SSE-KMS) is a feature not available yet

Hope it helps

lenglet-k commented 1 year ago

Hi, have you any news for this feature ?

lason-ovh commented 1 year ago

Hi, have you any news for this feature ?

Hi, as previously stated, this feature is due in Q3 2023

loliee commented 1 year ago

Hi, is this feature available in an alpha or beta version ? Thanks

biapar commented 10 months ago

Ready?

JustDoItSascha commented 8 months ago

Hi, have you any news for this feature ?

Hi, as previously stated, this feature is due in Q3 2023

I love how you made Q2 from @Izaia64 to Q3. And now we have Q4 and there is no update :(

agoude commented 8 months ago

Hi @JustDoItSascha, well thanks for your patience and highlighting the lighter side of the situation ;) The team has been pushing hard rebuilding the feature to make it better and ensuring the performance/resilience levels we wanna reach: the main reason behind those different shifts and we are sorry for such delays!

The wait has been longer than expected and we're working to make it worth your wait. Now, I can only say we never been so closed to deploy it and i'll keep you posted with a solid and hopefully final ETA 🙏!

JustDoItSascha commented 8 months ago

Ok cool, thank you :)

mahmoudaljan commented 7 months ago

@agoude Thank you for the hard work. We are eager to get our hands on the feature. I know that for us, this will make our lives much easier. Don't hesitate to share an ETA if you have any.

agoude commented 7 months ago

@agoude Thank you for the hard work. We are eager to get our hands on the feature. I know that for us, this will make our lives much easier. Don't hesitate to share an ETA if you have any.

hey @mahmoudaljan we all agree on this feature importance, no doubt! As I said earlier, we will shortly keep you posted with a solid ETA! thanks!

lason-ovh commented 6 months ago

Hi everyone, I wanted to ask: since the replication applies to only new objects i.e objects that are added AFTER the replication configuration is uploaded, we were wondering the importance of replicating the objects that already exist in the bucket.

on a scale of 1 (not important) to 5 (mandatory feature), what is for you the importance of providing a tool/mechanism to automatically replicate existing objects?
on a scale of 1 (I don't mind) to 5 (I'd hate to), should we provide that tool, do you mind executing a (very) few command lines?

biapar commented 6 months ago

+4 +4 plus by control panel.

mahmoudaljan commented 6 months ago

@lason-ovh

5 : You must assume that every client who needs this feature will look for ways to achieve this, better to deal with it on your own terms (bandwidth...) and limit complaints and questions due to client mishaps while trying to achieve this on their own.
2 : Since this will only be a procedure that will be executed once, no one will lack motivation to make it work at the beginning even if it requires a multi step procedure with command lines. Also, please do provide an API call if possible.

131 commented 6 months ago

5
2 // do whatever you want as long as terraform can apply it

lenglet-k commented 6 months ago

5
2

JustDoItSascha commented 5 months ago

5
5

fredleger commented 5 months ago

@lason-ovh

5 : You must assume that every client who needs this feature will look for ways to achieve this, better to deal with it on your own terms (bandwidth...) and limit complaints and questions due to client mishaps while trying to achieve this on their own.

2 : Since this will only be a procedure that will be executed once, no one will lack motivation to make it work at the beginning even if it requires a multi step procedure with command lines. Also, please do provide an API call if possible.

5
2 (as long as we can automate it in some way like script or jobs in our own infra)

mahmoudaljan commented 4 months ago

@agoude A little humor doesn't hurt.

https://giphy.com/gifs/fallontonight-jimmy-fallon-nbc-xUA7bcNrD7wxpIRgTS

agoude commented 4 months ago

@agoude A little humor doesn't hurt.

https://giphy.com/gifs/fallontonight-jimmy-fallon-nbc-xUA7bcNrD7wxpIRgTS

You're right and thanks for this!

lason-ovh commented 3 months ago

It's finally happening!!!! 😮 😮 The long awaited ASYNCHRONOUS REPLICATION will be very soon open in BETA (ETA = W14). You can already suscribe via the following link to be among our early users: https://survey.ovh.com/index.php/192786 Once the feature is activated, we will let you know. As always, we are keen on receiving your feedbacks, so do not hesitate! 😉

lason-ovh commented 2 months ago

Hi all, the Asynchronous Replication feature is now officially in beta!! 🥳 For those who signed up, you can now replicate your objects between buckets in the same region or in different regions. Check the official doc here: https://help.ovhcloud.com/csm/en-ie-public-cloud-storage-s3-asynchronous-replication-buckets?id=kb_article_view&sysparm_article=KB0062415 If you have questions and/or need help on this feature, do not hesitate to post a message on our Discord chan dedicated to S3

martonivan commented 2 months ago

Do you already have an estimated schedule about when the feature is expected to reach the 1.0 production phase?

ovh / public-cloud-roadmap

Object Storage - asynchronous replication #179