Iterating over a slice of strings (paths to files) and applying this to it

[robinryden]

Hello,

I have a question, how do I apply this to something where I iterate over a slice of strings (paths to files), since I'm also doing subtasks in the function i submit to the pool.

It looks something like this: File 1 -> fileHandler(client, path) -> Multipart upload to S3 (by streaming the files to a buffer, which also slices the file up into smaller pieces and at the end upload all the file parts as one unit.

thanks.

[alitto]

Hey @robinryden. Do you plan to use AWS SDK to do so? If you were to use the Upload Manager provided by AWS (https://aws.github.io/aws-sdk-go-v2/docs/sdk-utilities/s3/#upload-manager), note it would use its own internal worker pool to upload each file chuck, and it can be tuned to adjust chunk size (PartSize option) as well as the level of concurrency (Concurrency option). In this case, I'd simply iterate over the files and create an uploader for each of them:

// This pool could be instantiated once and reused
pool := pond.New(100, 1000)

// Create a task group for this specific group of files
group := pool.Group()

// Submit one task per file
for _, p := range filePaths {
    path := p
    group.Submit(func() {
    uploadFile(p) // uploadFile should take care of creating the Uploader
    })
}

// Wait for all uploads to complete
group.Wait()

You can control the number of concurrent uploaders by adjusting the size of the pool. You might have to test with different sizes to find the one that maximizes throughput. However, keep in mind there could be other factors affecting the actual level of concurrency you can achieve, like connection pool size on the S3 client, max connections limit on S3 servers, network quotas and so on.

[alitto]

Closing due to inactivity