Within the SDK, offer an io.Reader wrapper around SelectObjectContentEventStream. It doesn't have to do everything that SelectObjectContentEventStream does (i.e. metrics) as long as the differences are documented.
Use Case
When working with S3 Select, I frequently want to pass the output to functions that want a Reader (e.g. io.Copy). One specific AWS use case is to stream results of S3 Select back into S3 via s3manager where UploadInput accepts an io.Reader for the Body.
Proposed Solution
I could see this being implemented in the s3manager package as it's somewhat related. This is a GenAI implementation and I could imagine that there are error scenarios not handled here.
type s3SelectReader struct {
stream *s3.SelectObjectContentEventStream
remaining []byte // Buffer to store leftover data from previous event
closed bool // Flag indicating whether the reader has been closed
}
func (r *s3SelectReader) Read(b []byte) (n int, err error) {
// If the reader has been closed, return immediately with an error.
if r.closed {
return 0, io.ErrClosedPipe
}
var totalBytesRead int
for {
// If there is data remaining from the previous event, copy it to the output slice.
if len(r.remaining) > 0 {
n := copy(b[totalBytesRead:], r.remaining)
totalBytesRead += n
r.remaining = r.remaining[n:]
if totalBytesRead == len(b) {
return totalBytesRead, nil
}
}
data, ok := <-r.stream.Events()
if !ok {
r.closed = true
if totalBytesRead > 0 {
return totalBytesRead, nil
}
return 0, io.EOF
}
switch v := data.(type) {
case *s3types.SelectObjectContentEventStreamMemberRecords:
n := copy(b[totalBytesRead:], v.Value.Payload)
totalBytesRead += n
if n < len(v.Value.Payload) {
r.remaining = v.Value.Payload[n:]
}
if totalBytesRead == len(b) {
return totalBytesRead, nil
}
case *s3types.SelectObjectContentEventStreamMemberEnd:
//Update 4/25: Premature optimization to mark the stream as closed as soon as the event stream reaches an end.
//r.closed = true
if totalBytesRead > 0 {
return totalBytesRead, nil
}
return 0, io.EOF
default:
}
}
}
Other Information
No response
Acknowledgements
[X] I may be able to implement this feature request
Describe the feature
Within the SDK, offer an
io.Readerwrapper around SelectObjectContentEventStream. It doesn't have to do everything that SelectObjectContentEventStream does (i.e. metrics) as long as the differences are documented.Use Case
When working with S3 Select, I frequently want to pass the output to functions that want a Reader (e.g. io.Copy). One specific AWS use case is to stream results of S3 Select back into S3 via s3manager where UploadInput accepts an io.Reader for the Body.
Proposed Solution
I could see this being implemented in the s3manager package as it's somewhat related. This is a GenAI implementation and I could imagine that there are error scenarios not handled here.
Other Information
No response
Acknowledgements
AWS Go SDK V2 Module Versions Used
Go version used
go version go1.22.2 darwin/arm64