Understand if all necessary placeholders exist in Gateway-{ST,MT} for versioning to work

[amwolff]

Objective

The goal of this issue is to ensure that the gateway has all the necessary placeholders for handling the following S3 actions, to expedite the connection to the underlying Satellite actions when they become available for use.

Actions to verify

• PutBucketVersioning
  • Object versioning can be enabled for a bucket
  • Object versioning can be disabled for a bucket
  • When object versioning is disabled:
    • All existing versions are still maintained unless manually deleted
    • Any new versions will follow the behavior described below for PutObject
• GetBucketVersioning
  • The state of bucket version can be requested. The possible states are:
  • Enabled
  • Disabled
  • Suspended
• ListObjectVersions
  • Passes all relevant tests for ListObjectVersions
• GetObject
  • VersionId should be added as a request parameter to retrieve the specific object version
  • Response should include:
    • x-amz-version-id: Version of the object.
• HeadObject
  • VersionId should be added as a request parameter to retrieve the specific object version
  • Response includes:
    • x-amz-version-id: Version of the object.
• PutObject
  • For buckets with versioning enabled:
  • When an object that already exists (same name) is uploaded to a bucket:
    • A new copy of the object is stored with a new unique version ID
  • For buckets with versioning disabled:
  • When an object that already exists (same name) is uploaded to a bucket:
    • The latest version of the object is overwritten by the new object
    • The new object has a version ID of null
    • Versions older than the most recent version will not be impacted (this will only happen if versioning was previously enabled – in this instance they need to be manually deleted if desired)
  • Response includes:
  • x-amz-version-id: Version of the object.
• CompleteMultipartUpload
  • Same behavior as PutObject
  • Response includes:
  • x-amz-version-id: Version of the object.
• DeleteObject (need to validate against what has been implemented on Satellite)
  • A DeleteObject request should accept an optional VersionId, which references a specific version of the object
  • Response includes:
  • x-amz-version-id: The version ID of the delete marker created as a result of the DELETE operation
  • x-amz-delete-marker: Boolean that indicates whether the specified object version that was permanently deleted was (true) or was not (false) a delete marker before deletion. In a simple DELETE, this header indicates whether (true) or not (false) the current version of the object is a delete marker.
• DeleteObjects
  • Each Delete "Object" in a DeleteObjects request should accept an optional VersionId (in addition to the object Key), which references a specific version of the object
  • Response includes:
  • DeleteMarker: Boolean that indicates whether the specified object version that was permanently deleted was (true) or was not (false) a delete marker before deletion. In a simple DELETE, this header indicates whether (true) or not (false) the current version of the object is a delete marker.
  • DeleteMarkerVersionId: The version ID of the delete marker created as a result of the DELETE operation.
  • VersionId: The version ID of the deleted object.
• PutObjectTagging
  • VersionId should be added as a request parameter to specify the specific version of the object that the tag-set will be added to
  • Response includes:
  • x-amz-version-id: The versionId of the object the tag-set was added to.
• GetObjectTagging
  • VersionId should be added as a request parameter to retrieve the tagging information for a specific object version
  • Response includes:
    • x-amz-version-id: The versionId of the object for which you got the tagging information.
• DeleteObjectTagging
  • VersionId should be added as a request parameter to specify the specific versioned oibject to remove the tag-set from
  • Response includes:
    • x-amz-version-id: The versionId of the object the tag-set was removed from.

### Tasks
- [ ] PutBucketVersioning
- [ ] GetBucketVersioning
- [ ] ListObjectVersions
- [ ] GetObject
- [ ] HeadObject
- [ ] PutObject
- [ ] CompleteMultipartUpload
- [ ] DeleteObject
- [ ] DeleteObjects
- [ ] PutObjectTagging
- [ ] GetObjectTagging
- [ ] DeleteObjectTagging

[amwolff]

Jeremy had some feedback that what this task wants to accomplish is confusing, and I agree; the issue states that what should be done is verification if all placeholders exist while specifying what the expected behavior is. This list is a copy-paste from another issue, so that's probably why it's like that. Regardless, I think what we should do as part of this is verification if all placeholders exist along with data structures necessary for these actions to exhibit the expected behavior. So for example: DeleteObjectTagging: if the implementation of this action is already supported within our codebase, we should only verify if the code supports responding with something so that x-amz-version-id header will be returned (if there's nothing like that, maybe our underlying dependencies (minio) won't even respond with that header?). We should also verify if DeleteObjectTagging allows users to specify the version ID to be removed. If data structures of code for handling this action already include such a construct, we can be confident that clients will be able to communicate that, allowing this action to work the way we want here. @jewharton what do you think?

[jewharton]

@amwolff That sounds good to me. Thanks for clarifying.

[jewharton]

Evaluations were made based on the following versions:

• Gateway ST: storj/gateway-st@3a5764f
• Gateway MT: storj/edge@0c1c0d3
• MinIO: storj/minio@f1d4dd3

---

• PutBucketVersioning
  • :warning: Object versioning can be enabled/disabled for a bucket
    • Gateway MT: Yes
      • As of the time of writing, Gateway MT uses a version of Minio that doesn’t require the SetBucketVersioning method of Minio’s object layer interface to be implemented. Once the Minio dependency version is bumped to 730e38e or beyond, Gateway MT must implement this.
    • Gateway ST: No
• GetBucketVersioning
  • :warning: A bucket’s versioning status can be requested.
    • Gateway MT: Yes
      • As of the time of writing, Gateway MT uses a version of Minio that doesn’t require the GetBucketVersioning method of Minio’s object layer interface to be implemented. Once the Minio dependency version is bumped to 730e38e or beyond, Gateway MT must implement this.
    • Gateway ST: No
  • :warning: The possible states are:
    • Enabled (Supported)
    • Disabled (Unsupported; this state is only used by MFA delete, an unsupported feature)
    • Suspended (Supported)
• ListObjectVersions
  • :x: This action produces a "not implemented" error because the gateway’s single tenancy layer does not provide an implementation for it.
  • :warning: Passes all relevant tests for ListObjectVersions.
    • The test for this method passes, but it only verifies that the method returns a "not implemented" error.
• GetObject
  • :x: VersionId should be added as a request parameter to retrieve the specific object version.
    • Minio parses this value and passes it ([1], [2]) to the object API, but the gateway never uses it.
  • :x: Response should include x-amz-version-id, the version of the object.
    • Minio is capable of returning this value ([1], [2]), but the gateway never assigns it.
• HeadObject
  • :x: VersionId should be added as a request parameter to retrieve the specific object version.
    • Minio parses this value and passes it ([1], [2]) to the object API, but the gateway never uses it.
  • :x: Response includes x-amz-version-id, the version of the object.
    • Minio is capable of returning this value ([1], [2]), but the gateway never assigns it.
• PutObject
  • :x: There is no special handling for versioned objects in the gateway’s PutObject implementation. As a result, none of the following are true:
    • For buckets with versioning enabled:
      • When an object that already exists (same name) is uploaded to a bucket:
        • A new copy of the object is stored with a new unique version ID
    • For buckets with versioning disabled:
      • When an object that already exists (same name) is uploaded to a bucket:
        • The latest version of the object is overwritten by the new object
        • The new object has a version ID of null
        • Versions older than the most recent version will not be impacted (this will only happen if versioning was previously enabled – in this instance they need to be manually deleted if desired)
  • :x: Response includes x-amz-version-id, the version of the object.
    • Minio is capable of returning this value ([1], [2]), but the gateway never assigns it.
• CompleteMultipartUpload
  • :x: Same expected behavior as PutObject
    • There is no special handling for versioned objects in the gateway’s CompleteMultipartUpload implementation.
  • :x: Response includes x-amz-version-id, the version of the object.
    • Minio is capable of returning this value ([1], [2]), but the gateway never assigns it.
• DeleteObject (need to validate against what has been implemented on Satellite)
  • :x: VersionId should be added as a request parameter to delete a specific object version.
    • Minio uses this value in its API handler for this action and passes it ([1], [2]) to the object API, but the gateway never uses it.
  • :x: Response includes:
    • x-amz-version-id: The version ID of the delete marker created as a result of the DELETE operation
    • x-amz-delete-marker: Boolean that indicates whether the specified object version that was permanently deleted was (true) or was not (false) a delete marker before deletion. In a simple DELETE, this header indicates whether (true) or not (false) the current version of the object is a delete marker.
    • Minio is capable of returning these values ([1], [2]), but the gateway never assigns them.
• DeleteObjects (DeleteMultipleObjects)
  • :x: Each Delete "Object" in a DeleteObjects request should accept an optional VersionId (in addition to the object Key), which references a specific version of the object
    • Minio’s handler for this action supports this value and passes it to the gateway’s object API implementation, but it is never used.
  • :x: Response includes:
    • DeleteMarker: Boolean that indicates whether the specified object version that was permanently deleted was (true) or was not (false) a delete marker before deletion. In a simple DELETE, this header indicates whether (true) or not (false) the current version of the object is a delete marker.
    • DeleteMarkerVersionId: The version ID of the delete marker created as a result of the DELETE operation.
    • VersionId: The version ID of the deleted object.
    • Minio is capable of returning these values from its HTTP handler for this action, but the action’s implementation in gateway never assigns them.
• PutObjectTagging
  • :x: VersionId should be added as a request parameter to specify the specific version of the object that the tag-set will be added to.
    • Minio parses this value and passes it ([1], [2]) to the object API, but the gateway never uses it.
  • :x: Response includes x-amz-version-id, the version ID of the object the tag-set was added to.
    • Minio is capable of returning this value, but the gateway never assigns it.
• GetObjectTagging
  • :x: VersionId should be added as a request parameter to retrieve the tagging information for a specific object version.
    • Minio parses this value and passes it to the object API, but the gateway never uses it.
  • :x: Response includes x-amz-version-id, the version ID of the object for which you got the tagging information.
    • Minio is capable of returning this value, but the gateway never assigns it.
• DeleteObjectTagging
  • :x: VersionId should be added as a request parameter to specify the specific versioned object to remove the tag-set from.
    • Minio parses this value and passes it ([1], [2]) to the object API, but the gateway never uses it.
  • :x: Response includes x-amz-version-id, the version ID of the object the tag-set was removed from.
    • Minio is capable of returning this value, but the gateway never assigns it.