cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.77k stars 3.76k forks source link

backupccl: gs:// with AUTH=implicit hangs when metadata service unreachable #96373

Open dt opened 1 year ago

dt commented 1 year ago

Steps to Reproduce:

1) while running cockroach as not-root on a GCP machine, drop non-root traffic to the instance metadata service: sudo iptables -A OUTPUT -m owner ! --uid-owner root -d 169.254.169.254 -j DROP

2) SHOW BACKUPS IN 'gs://cockroach-fixtures?AUTH=implicit

Expected result: Some sort of error message saying that implicit auth failed

Actual result: Hangs forever

Version tested: v22.2.2

Stack of stuck SHOW:

goroutine 9158 [select]:
github.com/googleapis/gax-go/v2.Sleep({0x61c6a68, 0xc008438330}, 0x7f6733f14108?)
    github.com/googleapis/gax-go/v2/external/com_github_googleapis_gax_go_v2/invoke.go:57 +0x7b
cloud.google.com/go/internal.retry({0x61c6a68, 0xc008438330}, {0x3b9aca00, 0x6fc23ac00, 0x4000000000000000, 0x6fc23ac00}, 0xc003f13858, 0x561db10)
    cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go:49 +0x14d
cloud.google.com/go/internal.Retry(...)
    cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go:33
cloud.google.com/go/storage.run({0x61c6a68?, 0xc008438330?}, 0x52f8df2?, 0x9?, 0x0?)
    cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go:54 +0xce
cloud.google.com/go/storage.(*ObjectIterator).fetch(0xc0082d9ae0, 0x0, {0x0, 0x0})
    cloud.google.com/go/storage/external/com_google_cloud_go_storage/bucket.go:1558 +0xb7e
google.golang.org/api/iterator.(*PageInfo).fill(0xc00809b220, 0x40bde6f?)
    google.golang.org/api/iterator/external/org_golang_google_api/iterator/iterator.go:139 +0x33
google.golang.org/api/iterator.(*PageInfo).next(0xc00809b220)
    google.golang.org/api/iterator/external/org_golang_google_api/iterator/iterator.go:118 +0x9c
cloud.google.com/go/storage.(*ObjectIterator).Next(0xc0082d9ae0)
    cloud.google.com/go/storage/external/com_google_cloud_go_storage/bucket.go:1525 +0x29
github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).List(0xc004cabdc0, {0x61c6a68, 0xc007b20060}, {0x0?, 0x4618e7?}, {0x52de95e, 0x5}, 0xc004a6d210)
    github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:328 +0x246
github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest.ListFullBackupsInCollection({0x61c6a68, 0xc007b20060}, {0x6215b80, 0xc004cabe00})
    github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest/backup_destination.go:483 +0xc3
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.showBackupsInCollectionPlanHook.func1({0x61c6a68?, 0xc007b20030?}, {0x61c6a68?, 0xc00a521b30?, 0xf0f000000000000?}, 0x61c6a68?)
    github.com/cockroachdb/cockroach/pkg/ccl/backupccl/show.go:1337 +0x1d5
github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
    github.com/cockroachdb/cockroach/pkg/sql/planhook.go:199 +0xa4
created by github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec
    github.com/cockroachdb/cockroach/pkg/sql/planhook.go:197 +0x1b2

Jira issue: CRDB-24093

blathers-crl[bot] commented 1 year ago

cc @cockroachdb/disaster-recovery