A couple of people asked me to add a test type that would read pre-existing data from a bucket instead of running through the prepare step.
With this, we would potentially take the bucket_prefix as full bucket name, assume workers_share_buckets is True and in the prepare step scan for already existing files, then add them to the job queue.
Additionally I would add a existing_read_weight to the config that would weight the amount of reading existing files in the bucket. This way a mix of reading pre-existinng writes, listing & writing would still be possible...
A couple of people asked me to add a test type that would read pre-existing data from a bucket instead of running through the prepare step.
With this, we would potentially take the
bucket_prefixas full bucket name, assumeworkers_share_bucketsis True and in the prepare step scan for already existing files, then add them to the job queue.Additionally I would add a
existing_read_weightto the config that would weight the amount of reading existing files in the bucket. This way a mix of reading pre-existinng writes, listing & writing would still be possible...