roachtest: cache large files downloaded and used in roachtest tests

[jlinder]

Many of our roachtests download large files to use in the tests (some are 100MB+ in size). Add a way to cache these files locally / within cockroach infrastructure to reduce costs and load on the source systems.

One possible way to implement this:

1. create a google storage bucket
2. place the files we need there
3. download the files from the bucket
4. add some tracking mechanism to check for new versions of the files periodically (assuming they get updated) and automatically download the new file and place it in the bucket

If there are files loaded by the runs in AWS, add this to the above:

1. create an S3 bucket
2. place the files in it that are used by the tests run in S3
3. add a mechanism to roachtest that downloads the target files from the S3 bucket / gs bucket according to whether it's running in GCP or AWS
4. check for new versions of the files with the above defined tracking mechanism + placing the new versions in the S3 bucket

Epic DEVINF-109

Jira issue: CRDB-2900

[jlinder]

An example of these files: https://github.com/cockroachdb/cockroach/pull/56815

[andreimatei]

In the case of the Jepsen tests, what I think we want is to create an image with all the respective debian packages installed, not just to have a more reliable location to download them from.

[tbg]

Yeah, what we want here is an apt repo proxy. But I'll point out that the clouds already run them for us (if you're on recent distros). It's unlikely that we'll do a better job running them ourselves. So there may not be much to do here other than make sure we're staying on recent distros and verifying what I just said.

[andreimatei]

What I think we need is an image that already has all the packages installed. We shouldn't be downloading anything from any proxy.

[stevendanna]

We've recently discussed this over on the Test Eng team as well.

Speaking just of 3rd party resources that are currently downloaded from the internet, I think that we may want both an HTTP cache and custom images.

It would definitely be good to have artifacts that are used on a high proportion of roachtest runs to be baked into the image directly.

But, there are a number of benefits of also having our own repository of 3rd party resources:

• We have some use cases where it it would be nice to have faster and more reliable downloads without bloating the size of the images. For example, we may want to test on multiple versions of Kafka without bloating the images we use with all of those tests.

• Maintaining our own store of required test artifacts also allows us to continue to use those artifacts even if they've been removed by the upstream.

• It is nice for developers to be able to quickly add new versions of the artifact without having to also update the build image as part of your change.

• In addition to caching artifacts in an S3/GCS bucket, the caching code could store them on the local disk, which will speed up iteration for some test suites when run on "local" clusters.

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!

[stevendanna]

still relevant