phwissmann
commented
2 months ago Test application implemented:
- Multiple parallel uploads possible
- Upload rate: between 2Gbps and 3.6 Gbps, with md5 checksum
- Runs on Win (transfer computer) and Linux, as expected.
Description As a user I want to schedule uploads of my datasets by selecting a folder. Uploads should happen in parallel (or sequence) for different datasets. The upload should pick the next dataset(s) in the queue to be uploaded. The path within the s3 bucket can be specified.
Solution proposals Goroutines provide a good way to schedule tasks by using pipelines with Bounded parallelism. https://go.dev/blog/pipelines
Definition of Done Multiple folders can be selected by a frontend and the contents will be uploaded to a minio bucket.