amwolff commented 4 months ago

Goal

This issue is to make meaningful progress on Object Lock implementation within core and edge services. Specifically, to support immutable backups in backup software products like Veeam.

What needs to be done? • Acceptance Criteria

We only need to implement a small subset of Object Lock (for example, Legal Hold is out of scope at this moment) first. What needs to be implemented is defined in Object Lock milestone MVP section. To recap, it's:

What needs to be done is a test plan to

test fundamental function of aforementioned partial implementation
test S3-specific corner cases within the boundaries of the partial implementation
test Storj-specific corner cases within the boundaries of the partial implementation

Use a known template for test plans, if there is one.

Depends on…

This issue does not depend on anything and can be worked on immediately.

Links

an easy-to-read and insightful introduction to Object Lock: Using S3 Object Lock

neo-cypher commented 4 months ago

https://docs.google.com/document/d/1nxdeLJTJeIdgi3P35z_UQ9CkTN_MDUUIvrgsPYb6EUo/edit?usp=sharing

neo-cypher commented 4 months ago

Object Lock Test Plan

Objective

To validate the functionality, reliability, and performance of the object Lock feature to ensure it meets the requirements for object immutability and retention policies and only tests items within the scope of our implementation.

Scope

Testing will cover the Object Lock feature in compliance mode utilizing the retention period settings.
Security and permission tests to verify that only authorized users can alter the Object Lock settings.
- The new object lock permission should not be used with the existing read, write, list, or delete permissions. A more granular approach to permissions is out of scope.
New actions:
- GetObjectLockConfiguration: passes object_lock_enabled to entire system.
- PutObjectRetention: Waiting on verification in design document.
- GetObjectRetention: Waiting on verification in design document.
Update existing actions:
- CreateBucket: x-amz-bucket-object-lock-enabled needs to be handled.
- HeadObject: return lock information when user has Lock permissions
- GetObject: return lock information when user has Lock permissions.
- DeleteBucket: disallow deleting buckets with force when object_lock_enabled is true.
- DeleteProject: Prohibit deleting locked objects on project delete.

Test Cases:

Enable Object Lock:

Verify that Object Lock can be enabled on a new bucket.
Ensure Object Lock cannot be enabled on an existing bucket that didn't have it enabled at creation.
Only allow a bucket of versioned objects to be copied to buckets where object lock/versioning is enabled and configured approprately.
Verify that enabling object locks also enables object versioning.

Retention Periods:

Apply different retention periods to objects and verify that they cannot be deleted until the period expires.
Test changing the retention period and ensure it adheres to the mode's restrictions. Verify that the changes to configured retention periods are updating the restrictions on the file.
Apply overlapping retention periods to objects and verify they cannot be deleted.
Attempt to use the web ui and uplink to delete objects inside of the retention period.

New Actions:

GetObjectLockConfiguration
PutObjectRetention
GetObjectRetention

Modifications to existing endpoints:

CreateBucket: needs to pass object_lock_enabled to entire system.
HeadObject: only pass back lock related configuration information when the user making the request has the lock permission.
GetObject: needs to behave like HeadObject.
DeleteBucket: should fail gracefully while letting the customer know that the bucket has locked objects and cannot be deleted.
DeleteProject: should fail gracefully while letting the customer know that this project contains buckets which contain locked objects.

Legal Hold:

Verify any attempt to use legal hold services only returns an appropriate error.

Governance Mode

Verify any attempt to use governance mode services only returns an appropriate error.

Performance Testing

Data Access Performance:

Measure the time to upload, time to first byte, download, and access locked objects and compare it with non-locked objects to assess any performance impact.

Error Handling and Edge Cases

Invalid Operations:

Attempt to delete locked objects without required permissions and verify that errors are returned.
Try to enable object Lock on an existing bucket and ensure the operation fails.
Verify uplink and webui behavior as listed above and anywhere else where its fitting to use non s3 tools but to trigger s3 object log and holds.
Verify object lock enabled prohibits any users, including satellite admins, from deleting any locked objects from the network.

Conclusion

This test plan outlines a comprehensive approach to validate Object Lock feature's functionality, security, and performance. By executing this plan, we can ensure that the Object Lock feature works as intended, providing the necessary immutability and protection for stored objects.

kaloyan-raev commented 3 months ago

@neo-cypher A few things that should be reflected in the test plan:

Governance mode is outside the MVP scope. Any actions related to governance mode should return an appropriate error.
Same for legal hold.
Once enabled, object lock cannot be disabled on a bucket.
If a bucket is created with object lock enabled, the object versioning is also enabled for the bucket.
Suspending or disabling object versioning is not possible for a bucket with object lock enabled.
Force deletion is not possible for a bucket with object lock enabled.
Setting a default retention period for a bucket is outside of the MVP scope. Any attempt to do so should return an appropriate error.
The retention period on an object version can be extended, but it cannot be reduced.
We should have more specific test cases related to the new Lock permission.
The object lock is set on individual object versions, not on the object as a whole. if the current version of an object is locked, it should still be possible to upload newer versions on the same object and operate with them.

neo-cypher commented 3 months ago

@kaloyan-raev, I appreciate you taking the time to review this test plan. I had the scope incorrect so it skewed the requirements. Thanks for the corner cases, those are super helpful; as well as pointing out that when object lock is enabled that object versioning too is enabled and cannot be disabled or suspended. I'll get this feedback into the plan and update the scope to match our MVPs.