S3 Compatible Object Versioning - Beta

iglesiasbrandon commented 2 years ago

What is the problem/pain point?

Object Versioning is a key feature of S3-compatible object storage that Storj does not currently support. In order to attract and retain enterprise customers, object versioning needs to be supported.

What is Object Versioning?

Object versioning maintains multiple versions of every unique object in a bucket. Each time a new version of the same object is uploaded to the bucket, a new version is created. In addition, when an object is “deleted” a Delete Marker is added to the object rather than the execution of a hard delete. This allows for recovery of deleted objects in addition to recovery of previous versions of the object.

What is the impact?

Currently, Storj users have no way of keeping track of changes to objects in Storj DCS, and therefore customers have no ability to make use of the added benefits of versioning. Versioning would allow more control that users are already accustomed to having with other cloud storage providers and specifically enable use cases that require recovery from accidental deletion or overwriting of files.

Why now?

All major clouds (AWS, Google, Azure) support object versioning, so it is something all customers expect out of the box. As we begin to onboard enterprise customers with sophisticated use cases, closing this gap in S3 compatibility will bring us closer to our goal of being enterprise grade.

Links:

Milestones:

Pre-Requisite Work: https://github.com/storj/storj/milestone/46
Satellite: https://github.com/storj/storj/milestone/6
S3 Compatibility: https://github.com/storj/gateway-mt/milestone/4

PRD:

https://docs.google.com/document/u/1/d/1gZH5IWntGX-dcSsMSsgEeDGGQS-6emrppnksy9zZbkM/edit

Project information: https://storjlabs.atlassian.net/wiki/spaces/ENG/pages/2608431105/Object+Versioning

### Pre-Requisite Work
- [ ] https://github.com/storj/storj/issues/6045
- [ ] https://github.com/storj/storj/issues/6046
- [ ] https://github.com/storj/storj/issues/6047
- [ ] https://github.com/storj/storj/issues/6057
- [ ] https://github.com/storj/storj/issues/6048
- [ ] https://github.com/storj/storj/issues/6050
- [ ] https://github.com/storj/storj/issues/6255
- [ ] https://github.com/storj/storj/issues/6049
- [ ] https://github.com/storj/storj/issues/6054
- [ ] https://github.com/storj/storj/issues/6053
- [ ] https://github.com/storj/storj/issues/6051
- [ ] https://github.com/storj/storj/issues/6052

### Feature Development
- [ ] https://github.com/storj/storj/issues/5460
- [ ] https://github.com/storj/storj/issues/5808
- [ ] https://github.com/storj/storj/issues/6127

### S3 Compatibility
- [ ] https://github.com/storj/gateway-mt/issues/351
- [ ] https://github.com/storj/gateway-mt/issues/353
- [ ] https://github.com/storj/gateway-mt/issues/355
- [ ] https://github.com/storj/gateway-mt/issues/356

### Bucket-Level Versioning Configuration
- [ ] https://github.com/storj/storj/issues/6127
- [ ] https://github.com/storj/storj/issues/6385
- [ ] https://github.com/storj/storj/issues/6386
- [ ] https://github.com/storj/uplink/issues/155
- [ ] https://github.com/storj/storj/issues/6387

jtolio commented 2 years ago

Comment from OKR tracking sheet:

Need to solidify what features we're going to implement. Strong acceptance criteria required.

onionjake commented 2 years ago

@ferristocrat The metainfo team is getting to the point that they can help out with this milestone. We need to start filling out the requirements, etc.

iglesiasbrandon commented 1 year ago

Initial PRD is on a google doc: https://docs.google.com/document/u/1/d/1gZH5IWntGX-dcSsMSsgEeDGGQS-6emrppnksy9zZbkM/edit

storjrobot commented 1 year ago

This issue has been mentioned on Storj Community Forum (official). There might be relevant details there:

https://forum.storj.io/t/a-trash-for-customers/22513/3

ferristocrat commented 1 year ago

ROM ROM: 10-12 sprints

ferristocrat commented 12 months ago

@amwolff - let's talk through Object Versioning next steps from that gateway side when you're back. I think at this point we should prioritize this over UploadPartCopy related tasks.

iglesiasbrandon commented 11 months ago

@onionjake fyi the project information document has information we should discuss in our leadership meetings.

ferristocrat commented 10 months ago

Oct 27 2023: Moving out due to other business priorities that were worked on.

shaupt131 commented 8 months ago

Testing internally while we finish up a few priority fixes.

iglesiasbrandon commented 6 months ago

@amozoss do you think you could update our docs with information about object versioning?

ferristocrat commented 5 months ago

@amozoss do you think you could update our docs with information about object versioning?

I'm actually working on a doc. I'll create a PR this week in the docs repo

iglesiasbrandon commented 3 months ago

Closing this issue! Object versioning is in beta and any customer can enable it in the GUI.

storj / roadmap