search

opensearch-project / opensearch-benchmark

OpenSearch Benchmark - a community driven, open source project to run performance tests for OpenSearch
https://opensearch.org/docs/latest/benchmark/
Apache License 2.0
112 stars 79 forks source link

Add support for multi-part data corpora downloads #677

Closed gkamat closed 1 month ago

gkamat commented 1 month ago

Description

Permits data corpus files to be downloaded in parts. This is not intended for performance, but rather, to work around the restriction on file size that services like CloudFront might have.

Issues Resolved

543

Testing

Lint, unit and integ tests. Added a unit test to exercise the feature.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

gkamat commented 1 month ago

This looks good. Should we open an issue in the workloads repository for splitting the 1 TB file in Big5 workload?

Will just make the change directly.