search

google / go-cloud

The Go Cloud Development Kit (Go CDK): A library and tools for open cloud development in Go.
https://gocloud.dev/
Apache License 2.0
9.56k stars 810 forks source link

blob/gcsblob: TestConformance runs very slowly #2758

Closed jamie-digital closed 4 years ago

jamie-digital commented 4 years ago

Describe the bug

Testing blob/gcsblob runs for the full test timeout (10m by default) before giving up, due to a failed request. This may be caused by a corporate HTTP proxy not being used (and therefore the traffic being blocked).

To Reproduce

In module mode:

$ go get gocloud.dev/blob/gcsblob@v0.19.0
$ go test gocloud.dev/blob/gcsblob -v
Test logs 
=== RUN   TestConformance
=== RUN   TestConformance/TestList
=== RUN   TestConformance/TestList/no_objects
    TestConformance/TestList/no_objects: setup.go:98: Replaying from golden file testdata/TestConformance/TestList/no_objects.replay
2020/03/12 12:12:32 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS:0xc00035d080 Cancel: Response: ctx:}
2020/03/12 12:12:33 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:12:34 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:12:38 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:12:40 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:12:40 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:12:45 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:13:02 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:13:10 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:13:39 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:13:39 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:13:40 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:13:54 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:14:15 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:14:21 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:14:22 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:14:46 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:15:01 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:15:26 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
2020/03/12 12:15:27 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr:127.0.0.1:50902 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS: Cancel: Response: ctx:}
...
panic: test timed out after 10m0s

goroutine 102 [running]:
testing.(*M).startAlarm.func1()
    /Users/user/go/src/testing/testing.go:1460 +0xdf
created by time.goFunc
    /Users/user/go/src/time/sleep.go:168 +0x44

goroutine 1 [chan receive, 9 minutes]:
testing.(*T).Run(0xc000378240, 0x1808c22, 0xf, 0x18351b0, 0x108dac6)
    /Users/user/go/src/testing/testing.go:1044 +0x37e
testing.runTests.func1(0xc000378120)
    /Users/user/go/src/testing/testing.go:1285 +0x78
testing.tRunner(0xc000378120, 0xc00025fe10)
    /Users/user/go/src/testing/testing.go:992 +0xdc
testing.runTests(0xc0000c1d20, 0x1e9c4a0, 0x7, 0x7, 0x0)
    /Users/user/go/src/testing/testing.go:1283 +0x2a7
testing.(*M).Run(0xc000100680, 0x0)
    /Users/user/go/src/testing/testing.go:1200 +0x15f
main.main()
    _testmain.go:60 +0x135

goroutine 19 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000096820)
    /Users/user/dev/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
    /Users/user/dev/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:32 +0x57

goroutine 20 [chan receive, 9 minutes]:
testing.(*T).Run(0xc000378360, 0x1803c84, 0x8, 0xc0001c7600, 0x10)
    /Users/user/go/src/testing/testing.go:1044 +0x37e
gocloud.dev/blob/drivertest.RunConformanceTests(0xc000378240, 0x1835210, 0xc0001c75f0, 0x1, 0x1)
    /Users/user/dev/go/pkg/mod/gocloud.dev@v0.19.0/blob/drivertest/drivertest.go:186 +0x7e
gocloud.dev/blob/gcsblob.TestConformance(0xc000378240)
    /Users/user/dev/go/pkg/mod/gocloud.dev@v0.19.0/blob/gcsblob/gcsblob_test.go:102 +0x78
testing.tRunner(0xc000378240, 0x18351b0)
    /Users/user/go/src/testing/testing.go:992 +0xdc
created by testing.(*T).Run
    /Users/user/go/src/testing/testing.go:1043 +0x357

goroutine 21 [chan receive, 9 minutes]:
testing.(*T).Run(0xc000378480, 0x1805965, 0xa, 0xc00035abd0, 0x2)
    /Users/user/go/src/testing/testing.go:1044 +0x37e
gocloud.dev/blob/drivertest.testList(0xc000378360, 0x1835210)
    /Users/user/dev/go/pkg/mod/gocloud.dev@v0.19.0/blob/drivertest/drivertest.go:354 +0x7e0
gocloud.dev/blob/drivertest.RunConformanceTests.func1(0xc000378360)
    /Users/user/dev/go/pkg/mod/gocloud.dev@v0.19.0/blob/drivertest/drivertest.go:187 +0x34
testing.tRunner(0xc000378360, 0xc0001c7600)
    /Users/user/go/src/testing/testing.go:992 +0xdc
created by testing.(*T).Run
    /Users/user/go/src/testing/testing.go:1043 +0x357

goroutine 22 [select]:
github.com/googleapis/gax-go/v2.Sleep(0x1a01a80, 0xc0000b0008, 0x2962f6520, 0x0, 0x0)
    /Users/user/dev/go/pkg/mod/github.com/googleapis/gax-go/v2@v2.0.5/invoke.go:55 +0xd2
cloud.google.com/go/internal.retry(0x1a01a80, 0xc0000b0008, 0x3b9aca00, 0x6fc23ac00, 0x4000000000000000, 0x6fc23ac00, 0xc0006996c8, 0x1834f70, 0x10, 0x16eeac0)
    /Users/user/dev/go/pkg/mod/cloud.google.com/go@v0.54.0/internal/retry.go:47 +0xf1
cloud.google.com/go/internal.Retry(...)
    /Users/user/dev/go/pkg/mod/cloud.google.com/go@v0.54.0/internal/retry.go:31
cloud.google.com/go/storage.runWithRetry(0x1a01a80, 0xc0000b0008, 0xc0001ed8b8, 0xa, 0xc000202ec0)
    /Users/user/dev/go/pkg/mod/cloud.google.com/go/storage@v1.6.0/invoke.go:27 +0x87
cloud.google.com/go/storage.(*ObjectIterator).fetch(0xc00051c380, 0x3e8, 0x0, 0x0, 0x100e00c, 0xc0001eda00, 0x8, 0xc0000a3820)
    /Users/user/dev/go/pkg/mod/cloud.google.com/go/storage@v1.6.0/bucket.go:1166 +0x700
google.golang.org/api/iterator.(*PageInfo).fill(0xc0000970e0, 0x3e8, 0x16bf8e0, 0x1a0e001)
    /Users/user/dev/go/pkg/mod/google.golang.org/api@v0.20.0/iterator/iterator.go:139 +0x49
google.golang.org/api/iterator.(*Pager).NextPage(0xc0000a3830, 0x16bf8e0, 0xc00000fa80, 0x0, 0x0, 0xc0000a3830, 0xc00000fa60)
    /Users/user/dev/go/pkg/mod/google.golang.org/api@v0.20.0/iterator/iterator.go:216 +0x37e
gocloud.dev/blob/gcsblob.(*bucket).ListPaged(0xc00000fa60, 0x1a01a80, 0xc0000b0008, 0xc000097090, 0x30, 0x17d0940, 0x1)
    /Users/user/dev/go/pkg/mod/gocloud.dev@v0.19.0/blob/gcsblob/gcsblob.go:318 +0x2c7
gocloud.dev/blob.(*ListIterator).Next(0xc000699d98, 0x1a01a80, 0xc0000b0008, 0x0, 0x0, 0x0)
    /Users/user/dev/go/pkg/mod/gocloud.dev@v0.19.0/blob/blob.go:427 +0x245
gocloud.dev/blob/drivertest.iterToSetOfKeys(0x1a01a80, 0xc0000b0008, 0xc000378480, 0xc000699d98, 0xc00000fa60)
    /Users/user/dev/go/pkg/mod/gocloud.dev@v0.19.0/blob/drivertest/drivertest.go:902 +0x54
gocloud.dev/blob/drivertest.testList.func3(0xc000378480, 0x177d540, 0x0, 0x0)
    /Users/user/dev/go/pkg/mod/gocloud.dev@v0.19.0/blob/drivertest/drivertest.go:340 +0x30b
gocloud.dev/blob/drivertest.testList.func4(0xc000378480)
    /Users/user/dev/go/pkg/mod/gocloud.dev@v0.19.0/blob/drivertest/drivertest.go:355 +0x8f
testing.tRunner(0xc000378480, 0xc00035abd0)
    /Users/user/go/src/testing/testing.go:992 +0xdc
created by testing.(*T).Run
    /Users/user/go/src/testing/testing.go:1043 +0x357

goroutine 6 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x2096e38, 0x72, 0x0)
    /Users/user/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00051c298, 0x72, 0x0, 0x0, 0x180197a)
    /Users/user/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /Users/user/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc00051c280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/user/go/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc00051c280, 0x1, 0x0, 0xc000059ee8)
    /Users/user/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc00000f980, 0xc000059f10, 0x164b968, 0x0)
    /Users/user/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc00000f980, 0x1834e00, 0xc0000fe000, 0x1a082c0, 0xc0000ac480)
    /Users/user/go/src/net/tcpsock.go:261 +0x64
github.com/google/martian.(*Proxy).Serve(0xc0000fe000, 0x19ffd00, 0xc00000f980, 0x0, 0x0)
    /Users/user/dev/go/pkg/mod/github.com/google/martian@v2.1.1-0.20190517191504-25dcb96d9e51+incompatible/proxy.go:189 +0xb1
created by github.com/google/go-replayers/httpreplay/internal/proxy.(*Proxy).start
    /Users/user/dev/go/pkg/mod/github.com/google/go-replayers/httpreplay@v0.1.0/internal/proxy/record.go:145 +0x199

goroutine 8 [select]:
github.com/google/martian.(*Proxy).handle(0xc0000fe000, 0xc000020a40, 0x1a082c0, 0xc0000ac480, 0xc0000a3980, 0x0, 0x0)
    /Users/user/dev/go/pkg/mod/github.com/google/martian@v2.1.1-0.20190517191504-25dcb96d9e51+incompatible/proxy.go:271 +0x233
github.com/google/martian.(*Proxy).handleLoop(0xc0000fe000, 0x1a082c0, 0xc0000ac480)
    /Users/user/dev/go/pkg/mod/github.com/google/martian@v2.1.1-0.20190517191504-25dcb96d9e51+incompatible/proxy.go:250 +0x45d
created by github.com/google/martian.(*Proxy).Serve
    /Users/user/dev/go/pkg/mod/github.com/google/martian@v2.1.1-0.20190517191504-25dcb96d9e51+incompatible/proxy.go:218 +0x349

goroutine 52 [IO wait]:
internal/poll.runtime_pollWait(0x2096d58, 0x72, 0xffffffffffffffff)
    /Users/user/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00051c798, 0x72, 0xc00, 0xc6a, 0xffffffffffffffff)
    /Users/user/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /Users/user/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00051c780, 0xc000636000, 0xc6a, 0xc6a, 0x0, 0x0, 0x0)
    /Users/user/go/src/internal/poll/fd_unix.go:169 +0x201
net.(*netFD).Read(0xc00051c780, 0xc000636000, 0xc6a, 0xc6a, 0x203000, 0x110dd5b, 0xc00065e399)
    /Users/user/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000516018, 0xc000636000, 0xc6a, 0xc6a, 0x0, 0x0, 0x0)
    /Users/user/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc00000eb40, 0xc000636000, 0xc6a, 0xc6a, 0x0, 0xc00065a900, 0xc000619918)
    /Users/user/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc0002bd3d8, 0x19f2bc0, 0xc00000eb40, 0x100c325, 0x1739720, 0x17dd3a0)
    /Users/user/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc0002bd180, 0xacb10e8, 0xc000516018, 0x5, 0xc000516018, 0xc0000ac360)
    /Users/user/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc0002bd180, 0x0, 0x0, 0xc0006125a0)
    /Users/user/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
    /Users/user/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc0002bd180, 0xc00065e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /Users/user/go/src/crypto/tls/conn.go:1233 +0x15b
net/http.(*persistConn).Read(0xc0003785a0, 0xc00065e000, 0x1000, 0x1000, 0xc000619eb0, 0x10656e0, 0xc000619eb0)
    /Users/user/go/src/net/http/transport.go:1825 +0x75
bufio.(*Reader).fill(0xc00065a900)
    /Users/user/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc00065a900, 0x1, 0x2, 0x0, 0x0, 0xc000612500, 0x0)
    /Users/user/go/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc0003785a0)
    /Users/user/go/src/net/http/transport.go:1978 +0x1a8
created by net/http.(*Transport).dialConn
    /Users/user/go/src/net/http/transport.go:1647 +0xc56

goroutine 53 [select]:
net/http.(*persistConn).writeLoop(0xc0003785a0)
    /Users/user/go/src/net/http/transport.go:2277 +0x11c
created by net/http.(*Transport).dialConn
    /Users/user/go/src/net/http/transport.go:1648 +0xc7b

goroutine 117 [IO wait]:
internal/poll.runtime_pollWait(0x2096c78, 0x72, 0xffffffffffffffff)
    /Users/user/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00051c818, 0x72, 0x200, 0x205, 0xffffffffffffffff)
    /Users/user/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /Users/user/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00051c800, 0xc0001a8000, 0x205, 0x205, 0x0, 0x0, 0x0)
    /Users/user/go/src/internal/poll/fd_unix.go:169 +0x201
net.(*netFD).Read(0xc00051c800, 0xc0001a8000, 0x205, 0x205, 0xc000376a18, 0xc00061b9a0, 0x126b7b4)
    /Users/user/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000ac480, 0xc0001a8000, 0x205, 0x205, 0x0, 0x0, 0x0)
    /Users/user/go/src/net/net.go:184 +0x8e
io.(*multiReader).Read(0xc00000fb80, 0xc0001a8000, 0x205, 0x205, 0xc0001a8005, 0x0, 0x200)
    /Users/user/go/src/io/multi.go:26 +0xac
github.com/google/martian.(*peekedConn).Read(0xc00000fba0, 0xc0001a8000, 0x205, 0x205, 0xc0001a8000, 0x5, 0x10)
    /Users/user/dev/go/pkg/mod/github.com/google/martian@v2.1.1-0.20190517191504-25dcb96d9e51+incompatible/proxy.go:554 +0x52
crypto/tls.(*atLeastReader).Read(0xc0000c1e80, 0xc0001a8000, 0x205, 0x205, 0x15e, 0x200, 0xc00061baa0)
    /Users/user/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc0000b4cd8, 0x19f2bc0, 0xc0000c1e80, 0x100c325, 0x1739720, 0x1798e00)
    /Users/user/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc0000b4a80, 0xacb1108, 0xc00000fba0, 0x5, 0xc00000fba0, 0x14d)
    /Users/user/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc0000b4a80, 0x0, 0x0, 0x0)
    /Users/user/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
    /Users/user/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc0000b4a80, 0xc000193000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /Users/user/go/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).fill(0xc00007afc0)
    /Users/user/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).ReadSlice(0xc00007afc0, 0xa, 0x28, 0xc000506e78, 0x100e506, 0xc000105600, 0x100)
    /Users/user/go/src/bufio/bufio.go:359 +0x3d
bufio.(*Reader).ReadLine(0xc00007afc0, 0xc000506e80, 0xc00004cc00, 0x2011080, 0x0, 0x7, 0x30)
    /Users/user/go/src/bufio/bufio.go:388 +0x34
net/textproto.(*Reader).readLineSlice(0xc000078540, 0xc000105600, 0x0, 0x0, 0x0, 0x0)
    /Users/user/go/src/net/textproto/reader.go:58 +0x6c
net/textproto.(*Reader).ReadLine(...)
    /Users/user/go/src/net/textproto/reader.go:39
net/http.readRequest(0xc00007afc0, 0x1, 0xc000105600, 0x0, 0x0)
    /Users/user/go/src/net/http/request.go:1015 +0xa4
net/http.ReadRequest(...)
    /Users/user/go/src/net/http/request.go:1000
github.com/google/martian.(*Proxy).handle.func1(0xc0000a3980, 0xc00007bbc0, 0xc00007bb60)
    /Users/user/dev/go/pkg/mod/github.com/google/martian@v2.1.1-0.20190517191504-25dcb96d9e51+incompatible/proxy.go:264 +0x48
created by github.com/google/martian.(*Proxy).handle
    /Users/user/dev/go/pkg/mod/github.com/google/martian@v2.1.1-0.20190517191504-25dcb96d9e51+incompatible/proxy.go:263 +0x140
FAIL    gocloud.dev/blob/gcsblob    600.361s
FAIL

Expected behavior

Tests pass in a timely fashion. If connectivity isn't working, the test should give up after a limited number of attempts, rather than retrying until the test times out.

Version

v0.19.0

Additional context

This may be caused by a corporate HTTP proxy not being used (and therefore the traffic being blocked). Running on Go1.14 darwin amd64

vangent commented 4 years ago

The test should be hermetic (it uses record/replay). It works for me:

`` $ cd blob/gcsblob $ go test -v ... PASS ok gocloud.dev/blob/gcsblob 13.786s $

vangent commented 4 years ago

The URL in our record file for the first test is

https://www.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json\u0026delimiter=\u0026maxResults=1000\u0026pageToken=\u0026prefix=blob-for-list\u0026prettyPrint=false\u0026projection=full\u0026versions=false

compared to your error message

https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false

Some of those differences are due to encoding, but the most obvious delta is the hostname www.googleapis.com vs storage.googleapis.com.

Can you make sure you are using cloud.google.com/go v0.44.3? Maybe try starting with a fresh Go install. I think both of those hostnames will work in real life, but the record/replay is matching exact hostnames.

jamie-digital commented 4 years ago

When I run the tests from within the module, it works fine:

$ cd gocloud.dev/blob/gcsblob
$ go test -v
=== RUN   TestConformance
=== RUN   TestConformance/TestList
=== RUN   TestConformance/TestList/no_objects
    TestConformance/TestList/no_objects: setup.go:98: Replaying from golden file testdata/TestConformance/TestList/no_objects.replay
...
PASS

But when I run the tests from outside the module it fails:

$ cd mymodule
$ go test gocloud.dev/blob/gcsblob -v
=== RUN   TestConformance
=== RUN   TestConformance/TestList
=== RUN   TestConformance/TestList/no_objects
    TestConformance/TestList/no_objects: setup.go:98: Replaying from golden file testdata/TestConformance/TestList/no_objects.replay
2020/03/12 15:38:45 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body:<nil> GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:127.0.0.1:59175 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS:0xc0005b2000 Cancel:<nil> Response:<nil> ctx:<nil>}
2020/03/12 15:38:45 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body:<nil> GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:127.0.0.1:59175 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS:<nil> Cancel:<nil> Response:<nil> ctx:<nil>}
2020/03/12 15:38:47 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body:<nil> GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:127.0.0.1:59175 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS:<nil> Cancel:<nil> Response:<nil> ctx:<nil>}
2020/03/12 15:38:50 ERROR: martian: failed to round trip: no matching request for &{Method:GET URL:https://storage.googleapis.com/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[google-api-go-client/0.5 go-cloud/blob/0.1.0] X-Goog-Api-Client:[gl-go/1.14.0 gccl/20200228]] Body:<nil> GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:storage.googleapis.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:127.0.0.1:59175 RequestURI:/storage/v1/b/go-cloud-blob-test-bucket/o?alt=json&delimiter=&maxResults=1000&pageToken=&prefix=blob-for-list&prettyPrint=false&projection=full&versions=false TLS:<nil> Cancel:<nil> Response:<nil> ctx:<nil>}
...
FAIL

I just saw your second comment. I'm using cloud.google.com/go v0.54.0. Going back to v0.44.0 doesn't seem to change anything though. I'll try a fresh module.

jamie-digital commented 4 years ago

Dropping cloud.google.com/go from my go.mod completely resolved the issue, so I worked back through the versions looking for a change in behaviour. At cloud.google.com/go v0.52.0 the looping behaviour goes away but some of the tests fail:

--- FAIL: TestConformance (1.14s)
    --- PASS: TestConformance/TestList (0.45s)
        --- PASS: TestConformance/TestList/no_objects (0.40s)
        --- PASS: TestConformance/TestList/exactly_1_object_due_to_prefix (0.01s)
        --- PASS: TestConformance/TestList/no_pagination (0.01s)
        --- PASS: TestConformance/TestList/by_1 (0.01s)
        --- PASS: TestConformance/TestList/by_2 (0.01s)
        --- PASS: TestConformance/TestList/by_3 (0.01s)
        --- PASS: TestConformance/TestList/PaginationConsistencyAfterInsert (0.01s)
        --- PASS: TestConformance/TestList/PaginationConsistencyAfterDelete (0.01s)
    --- PASS: TestConformance/TestListWeirdKeys (0.01s)
    --- PASS: TestConformance/TestListDelimiters (0.04s)
        --- PASS: TestConformance/TestListDelimiters/fwdslash (0.01s)
        --- PASS: TestConformance/TestListDelimiters/backslash (0.01s)
        --- PASS: TestConformance/TestListDelimiters/abc (0.02s)
    --- FAIL: TestConformance/TestRead (0.07s)
        --- PASS: TestConformance/TestRead/read_of_nonexistent_key_fails (0.01s)
        --- PASS: TestConformance/TestRead/negative_offset_fails (0.00s)
        --- PASS: TestConformance/TestRead/length_0_read (0.01s)
        --- PASS: TestConformance/TestRead/read_from_positive_offset_to_end (0.01s)
        --- PASS: TestConformance/TestRead/read_a_part_in_middle (0.01s)
        --- PASS: TestConformance/TestRead/read_in_full (0.01s)
        --- FAIL: TestConformance/TestRead/read_in_full_with_negative_length_not_-1 (0.01s)
    --- PASS: TestConformance/TestAttributes (0.01s)
    --- FAIL: TestConformance/TestWrite (0.12s)
        --- PASS: TestConformance/TestWrite/write_to_empty_key_fails (0.01s)
        --- PASS: TestConformance/TestWrite/no_write_then_close_results_in_empty_blob (0.01s)
        --- PASS: TestConformance/TestWrite/no_write_then_close_results_in_empty_blob,_blob_existed (0.01s)
        --- PASS: TestConformance/TestWrite/invalid_ContentType_fails (0.00s)
        --- PASS: TestConformance/TestWrite/ContentType_is_discovered_if_not_provided (0.01s)
        --- PASS: TestConformance/TestWrite/write_with_explicit_ContentType_overrides_discovery (0.01s)
        --- FAIL: TestConformance/TestWrite/Content_md5_match (0.01s)
        --- PASS: TestConformance/TestWrite/Content_md5_did_not_match (0.01s)
        --- PASS: TestConformance/TestWrite/Content_md5_did_not_match,_blob_existed (0.01s)
        --- PASS: TestConformance/TestWrite/a_small_text_file (0.01s)
        --- PASS: TestConformance/TestWrite/a_large_jpg_file (0.01s)
        --- PASS: TestConformance/TestWrite/a_large_jpg_file_written_in_two_chunks (0.01s)
    --- PASS: TestConformance/TestCanceledWrite (0.03s)
        --- PASS: TestConformance/TestCanceledWrite/EmptyContentType (0.01s)
        --- PASS: TestConformance/TestCanceledWrite/NonEmptyContentType (0.01s)
        --- PASS: TestConformance/TestCanceledWrite/BlobExists (0.01s)
    --- PASS: TestConformance/TestConcurrentWriteAndRead (0.05s)
    --- PASS: TestConformance/TestMetadata (0.03s)
        --- PASS: TestConformance/TestMetadata/empty (0.01s)
        --- PASS: TestConformance/TestMetadata/empty_key_fails (0.00s)
        --- PASS: TestConformance/TestMetadata/duplicate_case-insensitive_key_fails (0.00s)
        --- PASS: TestConformance/TestMetadata/valid_metadata (0.01s)
        --- PASS: TestConformance/TestMetadata/valid_metadata_with_empty_body (0.01s)
        --- PASS: TestConformance/TestMetadata/valid_metadata_with_content_type (0.01s)
        --- PASS: TestConformance/TestMetadata/weird_metadata_keys (0.01s)
        --- PASS: TestConformance/TestMetadata/non-utf8_metadata_key (0.00s)
        --- PASS: TestConformance/TestMetadata/non-utf8_metadata_value (0.00s)
    --- PASS: TestConformance/TestMD5 (0.01s)
    --- PASS: TestConformance/TestCopy (0.02s)
        --- PASS: TestConformance/TestCopy/NonExistentSourceFails (0.00s)
        --- PASS: TestConformance/TestCopy/Works (0.01s)
    --- PASS: TestConformance/TestDelete (0.01s)
        --- PASS: TestConformance/TestDelete/NonExistentFails (0.00s)
        --- PASS: TestConformance/TestDelete/Works (0.01s)
    --- PASS: TestConformance/TestKeys (0.20s)
        --- PASS: TestConformance/TestKeys/non-UTF8_fails (0.00s)
        --- PASS: TestConformance/TestKeys/backslashes (0.01s)
        --- PASS: TestConformance/TestKeys/unicode (0.01s)
        --- PASS: TestConformance/TestKeys/ascii-5 (0.01s)
        --- PASS: TestConformance/TestKeys/dotdotbackslash (0.01s)
        --- PASS: TestConformance/TestKeys/spaces (0.01s)
        --- PASS: TestConformance/TestKeys/ascii-3 (0.01s)
        --- PASS: TestConformance/TestKeys/ascii-8 (0.01s)
        --- PASS: TestConformance/TestKeys/dotdotslash (0.01s)
        --- PASS: TestConformance/TestKeys/repeatedbackslashes (0.01s)
        --- PASS: TestConformance/TestKeys/startwithdigit (0.01s)
        --- PASS: TestConformance/TestKeys/fwdslashes (0.01s)
        --- PASS: TestConformance/TestKeys/repeatedfwdslashes (0.01s)
        --- PASS: TestConformance/TestKeys/quote (0.01s)
        --- PASS: TestConformance/TestKeys/ascii-1 (0.01s)
        --- PASS: TestConformance/TestKeys/ascii-2 (0.01s)
        --- PASS: TestConformance/TestKeys/ascii-4 (0.01s)
        --- PASS: TestConformance/TestKeys/ascii-6 (0.01s)
        --- PASS: TestConformance/TestKeys/ascii-7 (0.01s)
    --- PASS: TestConformance/TestSignedURL (0.05s)
    --- PASS: TestConformance/TestAs (0.03s)
        --- PASS: TestConformance/TestAs/verify_ContentLanguage_can_be_written_and_read_through_As (0.02s)
        --- PASS: TestConformance/TestAs/verify_As_returns_false_when_passed_nil (0.02s)

Going further back, the same issues continue until v0.44.0, where everything works again (not sure why it wasn't working when I made my previous comment). That would suggest cloud.google.com/go v0.45.0 breaks something and cloud.google.com/go v0.52.0 breaks it further.

Good thought on checking the cloud.google.com/go version!

vangent commented 4 years ago

I doubt that those cloud.google.com/go versions break anything if you use them live. They just changed some implementation details that happen to be recorded in the record/replay file. This kind of test is fragile against that kind of change.

If you want to dig deeper and/or confirm that, you can try running the tests with -record mode; you'll have to create a test bucket and update the code to point to it, and some env variables as well, so this is not super easy. https://github.com/google/go-cloud/blob/master/internal/docs/design.md#record-mode as more info.

jamie-digital commented 4 years ago

Would it be possible to modify the test to give up after a few attempts? I accept that the test is somewhat brittle, and I've set my CI pipeline to ignore this test's failures, but it feels a bit wasteful for it to keep retrying until it gets killed after 10 mins.