S3: `DeleteObjects` Creates File Versions and Delete Markers Regardless of File Existence with `ObjectLockEnabledForBucket` Set to True

kkb0318 commented 5 months ago

Acknowledgements

[X] I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
[X] I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

When creating a new S3 bucket with ObjectLockEnabledForBucket set to true, executing DeleteObjects on any resource name ( regardless of its existence ) results in the creation of some versions in the bucket. Consequently, I am unable to delete the bucket.

Expected Behavior

I expected no versions to be created since the file was never uploaded or otherwise created in the bucket.

Current Behavior

No errors occurred when executing the DeleteObjects operation, even though the specified objects did not exist. The output from the DeleteObjects call is as follows:

the code I write below, output shows like this

    output, err := client.DeleteObjects(ctx, &s3.DeleteObjectsInput{
        Bucket: bucketName,
        Delete: &types.Delete{Objects: objectIds},
    })
    log.Printf("%#v \n", output)

Output:

&s3.DeleteObjectsOutput{Deleted:[]types.DeletedObject{types.DeletedObject{DeleteMarker:(*bool)(0xc000014998), DeleteMarkerVersionId:(*string)(0xc000024ce0), Key:(*string)(0xc000024cd0), VersionId:(*string)(nil), noSmithyDocumentSerde:document.NoSerde{}}}, Errors:[]types.Error(nil), RequestCharged:"", ResultMetadata:middleware.Metadata{values:map[interface {}]interface {}{middleware.attemptSkewKey{}:233404000, middleware.rawResponseKey{}:(*http.Response)(0xc0000620a0), middleware.requestIDKey{}:"BDG9KGYH5SK4W8N3", middleware.responseAtKey{}:time.Time{wall:0xc17f11e52db153a0, ext:861094547, loc:(*time.Location)(0x8906660)}, middleware.serverTimeKey{}:time.Time{wall:0x0, ext:63848695573, loc:(*time.Location)(nil)}, retry.attemptResultsKey{}:retry.AttemptResults{Results:[]retry.AttemptResult{retry.AttemptResult{Err:error(nil), Retryable:false, Retried:false, ResponseMetadata:middleware.Metadata{values:map[interface {}]interface {}{middleware.attemptSkewKey{}:233404000, middleware.rawResponseKey{}:(*http.Response)(0xc0000620a0), middleware.requestIDKey{}:"BDG9KGYH5SK4W8N3", middleware.responseAtKey{}:time.Time{wall:0xc17f11e52db153a0, ext:861094547, loc:(*time.Location)(0x8906660)}, middleware.serverTimeKey{}:time.Time{wall:0x0, ext:63848695573, loc:(*time.Location)(nil)}, s3shared.hostID{}:"0+AeHUBI0JpOUs+WvT1l3axIdys78w/VKQuklMWRaNqzoSIRBjy4WCfqOb82r8fjh3K9UjKzwjc="}}}}}, checksum.computedInputChecksumsKey{}:map[string]string{"MD5":"YEIDfP5l6PMWu+KjCfq3dw=="}, s3shared.hostID{}:"0+AeHUBI0JpOUs+WvT1l3axIdys78w/VKQuklMWRaNqzoSIRBjy4WCfqOb82r8fjh3K9UjKzwjc="}}, noSmithyDocumentSerde:document.NoSerde{}}

Subsequent Error: When attempting to delete the bucket, an error occurred indicating that the bucket could not be deleted because it was not empty. The API error BucketNotEmpty was returned with a status code of 409, suggesting that all versions within the bucket need to be deleted first.

Attempting to retrieve an bucket results in the following error message: operation error S3: DeleteBucket, https response error StatusCode: 409, RequestID: xxxxxxxxxxxx, HostID: xxxxxxxxxxxxxxxxxxx, api error BucketNotEmpty: The bucket you tried to delete is not empty. You must delete all versions in the bucket.

Reproduction Steps

The value of bucketName should be changed to a unique name before running this code to avoid conflicts.

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
    "github.com/aws/aws-sdk-go-v2/service/s3/types"
)

func main() {
    fmt.Println("start")
    ctx := context.TODO()
    cfg, _ := config.LoadDefaultConfig(
        ctx,
    )
    client := s3.NewFromConfig(cfg)
    bucketName := aws.String("sample-bucket-xcvjofaf")
    region := "ap-northeast-1"

    // Attempt to create a new bucket with object lock enabled.
    _, err := client.CreateBucket(ctx, &s3.CreateBucketInput{
        Bucket: bucketName,
        CreateBucketConfiguration: &types.CreateBucketConfiguration{
            LocationConstraint: types.BucketLocationConstraint(region),
        },
        ObjectLockEnabledForBucket: aws.Bool(true),
    })
    if err != nil {
        log.Println(err)
    }

    // Attempt to delete an object that does not exist to test error handling.
    objectKeys := []string{"obj1"}
    objectIds := make([]types.ObjectIdentifier, len(objectKeys))
    for i, key := range objectKeys {
        objectIds[i] = types.ObjectIdentifier{Key: aws.String(key)}
    }
    _, err = client.DeleteObjects(ctx, &s3.DeleteObjectsInput{
        Bucket: bucketName,
        Delete: &types.Delete{Objects: objectIds},
    })
    if err != nil {
        log.Println(err)
    }

    // Attempt to delete the newly created bucket. Expected to delete, but [BucketNotEmpty] error occurred.
    _, err = client.DeleteBucket(ctx, &s3.DeleteBucketInput{
        Bucket: bucketName,
    })
    if err != nil {
        log.Println(err)
    }
}

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2 v1.26.1
github.com/aws/aws-sdk-go-v2/config v1.27.11
github.com/aws/aws-sdk-go-v2/service/s3 v1.53.1

Compiler and Version used

go version go1.22.2 darwin/amd64

Operating System and version

MacOS 14.4.1

RanVaknin commented 5 months ago

Hi @kkb0318 ,

This is the expected and documented behavior:

If bucket versioning is enabled, the operation inserts a delete marker, which becomes the current version of the object. To permanently delete an object in a versioned bucket, you must include the object’s versionId in the request. For more information about versioning-enabled buckets, see Deleting object versions from a versioning-enabled bucket.

"If bucket versioning is suspended, the operation removes the object that has a null versionId, if there is one, and inserts a delete marker that becomes the current version of the object. If there isn't an object with a null versionId, and all versions of the object have a versionId, Amazon S3 does not remove the object and only inserts a delete marker."

Thanks, Ran~

github-actions[bot] commented 5 months ago

This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

kkb0318 commented 4 months ago

@RanVaknin Apologies for the delayed response, and thank you for your answers.

I now clearly understand the cause, so I will close this issue.

github-actions[bot] commented 4 months ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

aws / aws-sdk-go-v2