Downloading GCS cache stalls

coryan commented 2 years ago

Seems like we need to configure a timeout for these downloads. It should complete in around 60 seconds when things are working well. A timeout of 5x would not be unreasonable.

2022-10-11T14:13:39Z (+0s): Running:
env -C '/h' /workspace/ci/cloudbuild/cache.sh restore --bucket_url=gs://cloud-cpp-testing-resources_cloudbuild/build-cache/google-cloud-cpp --key=main/fedora-36-quickstart-production/h --fallback_key=main/fedora-36-quickstart-production/h
2022-10-11T14:13:39Z (+0s): ====> cache.sh: restore
gs://cloud-cpp-testing-resources_cloudbuild/build-cache/google-cloud-cpp/main/fedora-36-quickstart-production/h/cache.tar.gz:
    Creation time:          Tue, 11 Oct 2022 13:27:54 GMT
    Update time:            Tue, 11 Oct 2022 13:27:54 GMT
    Storage class:          STANDARD
    Content-Length:         4585769035
    Content-Type:           application/x-tar
    Component-Count:        32
    Hash (crc32c):          aYEg+g==
    ETag:                   CJDK4PKj2PoCEAE=
    Generation:             1665494874072336
    Metageneration:         1
2022-10-11T14:13:41Z (+2s): Fetching cache url gs://cloud-cpp-testing-resources_cloudbuild/build-cache/google-cloud-cpp/main/fedora-36-quickstart-production/h/cache.tar.gz
Copying gs://cloud-cpp-testing-resources_cloudbuild/build-cache/google-cloud-cpp/main/fedora-36-quickstart-production/h/cache.tar.gz to file:///tmp/tmp.S2dl2lyXhQ/cache.tar.gz

This failed before we created public logs:

https://pantheon.corp.google.com/cloud-build/builds;region=us-east1/a148579a-ec23-4055-89cd-0eed715a67eb;step=2?mods=logs_tg_prod&project=cloud-cpp-testing-resources