aws / aws-sdk-go-v2

AWS SDK for the Go programming language.
https://aws.github.io/aws-sdk-go-v2/docs/
Apache License 2.0
2.5k stars 602 forks source link

Adding support of versionId in s3:SelectObjectContent(short issue description) #2585

Closed mappie-grofers closed 3 months ago

mappie-grofers commented 3 months ago

Describe the feature

Ideally select object content should be similar to getObject except it does select on conent. Currently it lacks VersionId support in params. There is a way to specify version id in s3 select as per this github issue: https://github.com/aws/aws-sdk-java/issues/2357#issuecomment-644896382

Use Case

We only want to fetch the required content hence using selectObjectContent. But due to this we are not able to select content from older versions. Please add VersionId support in SelectObjectContent

Proposed Solution

Adding support of versionId as it is in GetObject

Other Information

There is a way to specify version id in s3 select as per this github issue: https://github.com/aws/aws-sdk-java/issues/2357#issuecomment-644896382

Acknowledgements

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2/service/s3 v1.40.2

Go version used

1.21

lucix-aws commented 3 months ago

@mappie-grofers --

Based on the Java issue you've provided, the way this is done in the Java SDK is actually through generic HTTP request manipulation. This same level of customization is possible in the Go V2 SDK as well through middleware.

The following example demonstrates how to do this:

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
    "github.com/aws/smithy-go/middleware"
    smithyhttp "github.com/aws/smithy-go/transport/http"
)

type withQueryParam struct {
    key, value string
}

var _ middleware.SerializeMiddleware = (*withQueryParam)(nil)

func (*withQueryParam) ID() string { return "withQueryParam" }

func (m *withQueryParam) HandleSerialize(
    ctx context.Context, in middleware.SerializeInput, next middleware.SerializeHandler,
) (
    out middleware.SerializeOutput, md middleware.Metadata, err error,
) {
    req, ok := in.Request.(*smithyhttp.Request)
    if !ok {
        return out, md, fmt.Errorf("unexpected transport %T", in.Request)
    }

    // smithyhttp.Request embeds http.Request
    req.URL.RawQuery = fmt.Sprintf("%s&%s=%s", req.URL.RawQuery, m.key, m.value)
    return next.HandleSerialize(ctx, in)
}

// you don't _need_ to have this but it makes adding the middleware to multiple
// operations much easier
func addWithQueryParam(key, value string) func(*s3.Options) {
    return func(o *s3.Options) {
        o.APIOptions = append(o.APIOptions, func(s *middleware.Stack) error {
            return s.Serialize.Add(&withQueryParam{key, value}, middleware.After)
        })
    }
}

func main() {
    cfg, err := config.LoadDefaultConfig(context.Background())
    if err != nil {
        log.Fatal(err)
    }

    svc := s3.NewFromConfig(cfg, func(o *s3.Options) {
        o.ClientLogMode = aws.LogRequest
    })

    svc.GetObject(context.Background(), &s3.GetObjectInput{
        Bucket: aws.String("bucket"),
        Key:    aws.String("key"),
    }, addWithQueryParam("foo", "bar"))
}

This is just with GetObject as an example, you can apply this to any S3 operation using functional options like I've done here.

Please try that and let us know if it works for you.

lucix-aws commented 3 months ago

Aside: if this is something the S3 API can do, it should really be part of their API model such that all SDKs pick it up automatically, at which point it would be a field in the request input like any other parameter. Unsure why that's not the case.

Tracking internally: V1320702086

github-actions[bot] commented 3 months ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

lucix-aws commented 2 months ago

S3 select has acknowledged this and created a backlog item to address the modeling defect.

mappie-grofers commented 2 months ago

Thank you for the information. Please keep us posted. Regards Mahesh Paliwal

On Thu, 11 Apr 2024 at 9:17 PM, Luc Talatinian @.***> wrote:

S3 select has acknowledged this and created a backlog item to address the modeling defect.

— Reply to this email directly, view it on GitHub https://github.com/aws/aws-sdk-go-v2/issues/2585#issuecomment-2050001952, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3RO56POWDRPRAPAZ5YK2UTY42WBLAVCNFSM6AAAAABFMJBQK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJQGAYDCOJVGI . You are receiving this because you were mentioned.Message ID: @.***>

--

This email is intended only for the person or the entity to whom it is addressed. If you are not the intended recipient, please delete this email and contact the sender.