Review Ceph's test suite to find potential improvements to requirements for Object Versioning

amwolff commented 11 months ago

Motivation/Why?

Our integration test suites are quite good at finding edge cases in certain features of S3. We should review them to find any potential edge cases that PRD doesn't list as requirements.

What needs to be done?

Review object versioning tests from Ceph's test suite to find any potential edge cases. Verify them against the Requirements section of this document.

Acceptance Criteria

Additional edge cases are reported in the document as suggestions or comments, and product managers are aware of changes that need to be made to the requirements documents.

Links

amwolff commented 10 months ago

from sprint planning: it might be worth running @jtolio's object versioning wrapper (https://review.dev.storj.io/c/storj/gateway/+/10925) against Ceph's test suite and see what edge cases were missed (if any).

halkyon commented 9 months ago

I ran the splunk test suite.

I removed the skip_for_storj we had on versioning tests to force those to run.
- GetBucketVersioning isn't supported, so I removed the checks for those and checking bucket versioning status enabled (so it's assuming enabled) to allow tests to continue running.
- Versioning suspended tests can't be tested because we don't support suspended state, so those are still ignored.
- One test ensures that if versioning state is suspended, no version IDs should be returned for new objects. Maybe that's obvious for the implementation, but it might be worth pointing out.
Some list failures (regressions, these work on the existing gateway):
- One of them tried to use a list marker but it didn't return the correct truncated object list.
- Another wanted a key name like "obj1" but got "obj1#D1RQF92QMBAK1NIN4G678DCIDG" (the version suffixed to it), so maybe there's some missing places where it rewrites the object keys back from the versioned ones (?)
ListMultipartUploads not supported, so that test fails. I guess that needs to do the thing like ListObjects where it processes the keys that include version suffix.
- It might be worth specifically making sure multi-part uploads still work with versioning, e.g. making sure CompleteMultipartUpload works the same as PutObject in that it creates a new version.
Most of the version tests are failing on a 405 Method Not Allowed from the gateway. The shim is returning that if it encounters a delete marker.

HEAD /s3-tests-2fgslb1zpljz5zu2bpq2-1/testobj HTTP/1.1
Host: localhost:7777
Accept-Encoding: identity
Date: Thu, 28 Sep 2023 04:30:36 GMT
Content-Length: 0
Authorization: AWS zjc8QPFzK1FHJpyZUwP6UHkFd4r:efFsacq7UiJsYno5KsvHsFn3j1s=
User-Agent: Boto/2.49.0 Python/2.7.18 Linux/6.2.0-33-generic

HTTP/1.1 405 Method Not Allowed
Accept-Ranges: bytes
Content-Length: 0
Content-Security-Policy: block-all-mixed-content
Server: Storj
Vary: Origin
X-Amz-Request-Id: 1788F5B976B35144
X-Xss-Protection: 1; mode=block
Date: Thu, 28 Sep 2023 04:30:36 GMT

All the multipart tests errored out, that's a big regression (!). I noticed an error in the logs:

2023-09-28T16:18:00+13:00       ERROR   miniogw/single_tenant.go:97     error:  {"error": "Bad digest: Expected 37b51d194a7513e45b56f6524f2d51f2 does not match calculated 07159c47ee1b19ae4fb9c40d480856c4"}

The client saw this error, but I don't immediately know what was going on that would cause that:

<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Key>mymultipart</Key><BucketName>s3-tests-n77lfnt5uh6riaf8yhy3-110</BucketName><Resource>/s3-tests-n77lfnt5uh6riaf8yhy3-110/mymultipart</Resource><RequestId>1788F2370700527F</RequestId><HostId>98d06a5b-f794-4d84-bd62-56d261468ac3</HostId></Error>

The patched tests can be run with the following (from my Docker user):

docker run --rm --add-host=localhost:host-gateway -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e ENDPOINT=localhost:7777 -e SECURE=0 halkyon/splunk-s3-tests:patched-versioning

Here's the full test run results including failures: https://gist.github.com/halkyon/fa409f08900f8ed1acad8ec32d2fafe1

amwolff commented 9 months ago

@ferristocrat will follow up with @shaupt131 on validating results vs. requirements being implemented

amwolff commented 8 months ago

I think we can close this issue now.

storj / edge