search

googleapis / google-cloud-go

Google Cloud Client Libraries for Go.
https://cloud.google.com/go/docs/reference
Apache License 2.0
3.73k stars 1.28k forks source link

build: fix failing builds #335

Closed broady closed 8 years ago

broady commented 8 years ago

It's been failing for over a week https://travis-ci.org/GoogleCloudPlatform/gcloud-golang/builds

/cc @zombiezen @jba @adams-sarah @shantuo

broady commented 8 years ago

I enabled the Vision API on the project. The other failures are real test failures or flaky tests that should be addressed.

s-mang commented 8 years ago

We should make CI integration with Gerrit a priority. I'm happy to finish your work on this @broady once I finish datastore entity_values.

broady commented 8 years ago

Awesome. For reference - https://github.com/broady/travis-tools

My suggestion is using the Container Builder API to execute the tests. https://cloud.google.com/container-builder/docs/quickstart

Sample implementation of that API: https://github.com/broady/cdbuild/blob/master/main.go

Not sure about integration with Gerrit (ala Go's TryBot), but I think it works via web hooks.

/cc @bradfitz

s-mang commented 8 years ago

Awesome @broady I'll start on this ASAP.

As for the failing builds, seems to be consistently a bigtable test failing:

--- FAIL: TestClientIntegration (10.05s)
    bigtable_test.go:81: bttest.Server running on 127.0.0.1:39909
    bigtable_test.go:68: [dialed Client] 532.026µs since start, 532.026µs since last checkpoint
    bigtable_test.go:68: [dialed AdminClient] 587.202µs since start, 55.176µs since last checkpoint
    bigtable_test.go:68: [created table] 2.446517ms since start, 1.859315ms since last checkpoint
    bigtable_test.go:68: [created "follows" column family] 3.721189ms since start, 1.274672ms since last checkpoint
    bigtable_test.go:68: [inserted initial data] 7.500479ms since start, 3.77929ms since last checkpoint
    bigtable_test.go:68: [did two conditional mutations] 10.341037ms since start, 2.840558ms since last checkpoint
    bigtable_test.go:68: [tested ReadRow] 11.962897ms since start, 1.62186ms since last checkpoint
    bigtable_test.go:68: [tested ReadRows in a few ways] 23.252176ms since start, 11.289279ms since last checkpoint
    bigtable_test.go:68: [did partial ReadRows test] 25.019009ms since start, 1.766833ms since last checkpoint
    bigtable_test.go:68: [exercised DeleteRow] 27.411282ms since start, 2.392273ms since last checkpoint
    bigtable_test.go:68: [tested ReadModifyWrite] 31.997901ms since start, 4.586619ms since last checkpoint
    bigtable_test.go:68: [tested multiple versions in a cell] 38.662771ms since start, 6.66487ms since last checkpoint
    bigtable_test.go:68: [tested high concurrency] 172.299278ms since start, 133.636507ms since last checkpoint
    bigtable_test.go:489: Doing large scan: rpc error: code = 4 desc = context deadline exceeded
    bigtable_test.go:492: Large scan returned 10579968 bytes, want 83968000
    bigtable_test.go:502: Scan with row limit returned 0 rows, want 3
    bigtable_test.go:68: [tested big read/write/scan] 10.052201816s since start, 9.879902538s since last checkpoint
    bigtable_test.go:508: Creating column family: rpc error: code = 4 desc = context deadline exceeded

This test has been failing since the "turn on race detector" landed.

garye commented 8 years ago

I'll make a change to loosen the deadline for that test.

On Tue, Aug 30, 2016, 5:00 PM Sarah Adams notifications@github.com wrote:

Awesome @broady https://github.com/broady I'll start on this ASAP.

As for the failing builds, seems to be consistently a bigtable test failing:

--- FAIL: TestClientIntegration (10.05s) bigtable_test.go:81: bttest.Server running on 127.0.0.1:39909 bigtable_test.go:68: [dialed Client] 532.026µs since start, 532.026µs since last checkpoint bigtable_test.go:68: [dialed AdminClient] 587.202µs since start, 55.176µs since last checkpoint bigtable_test.go:68: [created table] 2.446517ms since start, 1.859315ms since last checkpoint bigtable_test.go:68: [created "follows" column family] 3.721189ms since start, 1.274672ms since last checkpoint bigtable_test.go:68: [inserted initial data] 7.500479ms since start, 3.77929ms since last checkpoint bigtable_test.go:68: [did two conditional mutations] 10.341037ms since start, 2.840558ms since last checkpoint bigtable_test.go:68: [tested ReadRow] 11.962897ms since start, 1.62186ms since last checkpoint bigtable_test.go:68: [tested ReadRows in a few ways] 23.252176ms since start, 11.289279ms since last checkpoint bigtable_test.go:68: [did partial ReadRows test] 25.019009ms since start, 1.766833ms since last checkpoint bigtable_test.go:68: [exercised DeleteRow] 27.411282ms since start, 2.392273ms since last checkpoint bigtable_test.go:68: [tested ReadModifyWrite] 31.997901ms since start, 4.586619ms since last checkpoint bigtable_test.go:68: [tested multiple versions in a cell] 38.662771ms since start, 6.66487ms since last checkpoint bigtable_test.go:68: [tested high concurrency] 172.299278ms since start, 133.636507ms since last checkpoint bigtable_test.go:489: Doing large scan: rpc error: code = 4 desc = context deadline exceeded bigtable_test.go:492: Large scan returned 10579968 bytes, want 83968000 bigtable_test.go:502: Scan with row limit returned 0 rows, want 3 bigtable_test.go:68: [tested big read/write/scan] 10.052201816s since start, 9.879902538s since last checkpoint bigtable_test.go:508: Creating column family: rpc error: code = 4 desc = context deadline exceeded

This test has been failing since the "turn on race detector" landed.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/gcloud-golang/issues/335#issuecomment-243579173, or mute the thread https://github.com/notifications/unsubscribe-auth/AACTr9sfTFHyKqgYc_8DLEt7E4fPEBUpks5qlJnXgaJpZM4Jw9Fr .

shantuo commented 8 years ago

I cannot reproduce the pubsub test failure. I ran multiple times for all the pubsub tests with race flag when I fixed the pubsub unit tests and never saw this one.

--- FAIL: TestAll (7.20s)
    integration_test.go:71: CreateSub error: googleapi: Error 503: Authentication backend unavailable., backendError
    integration_test.go:87: subscription subscription-1472558298 should exist, but it doesn't
    integration_test.go:121: error constructing iterator: googleapi: Error 404: Resource not found (resource=subscription-1472558298)., notFound
shantuo commented 8 years ago

I am working on the storage test failure:


--- FAIL: TestCondition (0.06s)
    storage_test.go:395: 5. RequestURI = "GET /storage/v1/b/buck/o/obj?alt=json&ifMetagenerationNotMatch=1234&projection=full"; want "GET https://www.googleapis.com/storage/v1/b/buck/o/obj?alt=json&ifMetagenerationNotMatch=1234&projection=full"
    storage_test.go:395: 6. RequestURI = "PATCH /storage/v1/b/buck/o/obj?alt=json&ifMetagenerationMatch=1234&projection=full"; want "PATCH https://www.googleapis.com/storage/v1/b/buck/o/obj?alt=json&ifMetagenerationMatch=1234&projection=full"
    storage_test.go:395: 7. RequestURI = "DELETE /storage/v1/b/buck/o/obj?alt=json&generation=1234"; want "DELETE https://www.googleapis.com/storage/v1/b/buck/o/obj?alt=json&generation=1234"
    storage_test.go:395: 8. RequestURI = "POST /upload/storage/v1/b/buck/o?alt=json&ifGenerationMatch=1234&projection=full&uploadType=multipart"; want "POST https://www.googleapis.com/upload/storage/v1/b/buck/o?alt=json&ifGenerationMatch=1234&projection=full&uploadType=multipart"
    storage_test.go:395: 9. RequestURI = "POST /storage/v1/b/buck/o/obj/copyTo/b/dstbuck/o/dst?alt=json&ifMetagenerationMatch=5678&ifSourceGenerationMatch=1234&projection=full"; want "POST https://www.googleapis.com/storage/v1/b/buck/o/obj/copyTo/b/dstbuck/o/dst?alt=json&ifMetagenerationMatch=5678&ifSourceGenerationMatch=1234&projection=full"
bradfitz commented 8 years ago

@shantuo, that latest one is related to @okdave's recent change: https://github.com/google/google-api-go-client/commit/bb198ac8a40e08503277ebd527a8898bbfbe27fb

You should just be able to remove the https://www.googleapis.com/storage part of the "want" values.

shantuo commented 8 years ago

@bradfitz Thanks for the pointer!

shantuo commented 8 years ago

For the failure in the new logging package we got:

2016/08/30 11:57:06 creating prod client: google.DefaultTokenSource: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
FAIL    cloud.google.com/go/preview/logging 0.049s

Do I need some special credentials for this preview/logging API?

broady commented 8 years ago

Sounds like a bad test. Check the other tests to see how to create a TokenSource suitable for integration tests.

jba commented 8 years ago

No special creds. Should work if you've configured your environment as per CONTRIBUTING.md.

On Tue, Aug 30, 2016 at 8:03 PM Chris Broadfoot notifications@github.com wrote:

Sounds like a bad test. Check the other tests to see how to create a TokenSource suitable for integration tests.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/gcloud-golang/issues/335#issuecomment-243619559, or mute the thread https://github.com/notifications/unsubscribe-auth/ARoHZb537RQWlZEbo3BiBdniu2efiX8Uks5qlMTigaJpZM4Jw9Fr .

broady commented 8 years ago

Sent CL 7271 for the logging credentials. option.WithGRPCDialOption doesn't override the application default credentials.

Basically, no one should ever use option.WithGRPCDialOption(grpc.WithPerRPCCredentials(...)).

Also, WithClientTLSFromCert(nil, "") is the default.

broady commented 8 years ago

Logging is better now, but still fails:

Sink: name:"GO-CLIENT-TEST-SINK-1472621764-0013" destination:"storage.googleapis.com/dulcet-port-762" filter:"logName = \"projects/dulcet-port-762/logs/GO-LOGGING-CLIENT-TEST-LOG\"" output_version_format:V2 
--- FAIL: TestCreateDeleteSink (0.11s)
    sinks_test.go:72: rpc error: code = 7 desc = The caller does not have permission
=== RUN   TestUpdateSink
--- FAIL: TestUpdateSink (0.08s)
    sinks_test.go:107: rpc error: code = 7 desc = The caller does not have permission
=== RUN   TestListSinks
Sink: name:"GO-CLIENT-TEST-SINK-1472621764-0015" destination:"storage.googleapis.com/dulcet-port-762" filter:"logName = \"projects/dulcet-port-762/logs/GO-LOGGING-CLIENT-TEST-LOG\"" output_version_format:V2 
--- FAIL: TestListSinks (0.08s)
    sinks_test.go:150: Create("GO-CLIENT-TEST-SINK-1472621764-0015"): rpc error: code = 7 desc = The caller does not have permission

@jba - that bucket has given the logs service account Owner permission. What's this failure? I reproduce this locally with my cbro-scratch project, too.

edit: added cloud-logs@google.com as an editor (as per the docs) but it still fails with the same error.

jba commented 8 years ago

Chris,

First let's fix your local project.

Did you do

$ gsutil acl ch -u cloud-logs@system.gserviceaccount.com:O gs://cbro-scratch

On Wed, Aug 31, 2016 at 1:43 AM Chris Broadfoot notifications@github.com wrote:

Logging is better now, but still fails:

Sink: name:"GO-CLIENT-TEST-SINK-1472621764-0013" destination:"storage.googleapis.com/dulcet-port-762" filter:"logName = \"projects/dulcet-port-762/logs/GO-LOGGING-CLIENT-TEST-LOG\"" output_version_format:V2 --- FAIL: TestCreateDeleteSink (0.11s) sinks_test.go:72: rpc error: code = 7 desc = The caller does not have permission === RUN TestUpdateSink --- FAIL: TestUpdateSink (0.08s) sinks_test.go:107: rpc error: code = 7 desc = The caller does not have permission === RUN TestListSinks Sink: name:"GO-CLIENT-TEST-SINK-1472621764-0015" destination:"storage.googleapis.com/dulcet-port-762" filter:"logName = \"projects/dulcet-port-762/logs/GO-LOGGING-CLIENT-TEST-LOG\"" output_version_format:V2 --- FAIL: TestListSinks (0.08s) sinks_test.go:150: Create("GO-CLIENT-TEST-SINK-1472621764-0015"): rpc error: code = 7 desc = The caller does not have permission

@jba https://github.com/jba - that bucket has given the logs service account Owner permission. What's this failure? I reproduce this locally with my cbro-scratch project, too.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/gcloud-golang/issues/335#issuecomment-243664208, or mute the thread https://github.com/notifications/unsubscribe-auth/ARoHZWczrQeERxh_pvI0fv1KpvDkWU7sks5qlRR2gaJpZM4Jw9Fr .

broady commented 8 years ago

Yup, I have those permissions. Same failures.

Also ran gsutil acl ch -g cloud-logs@google.com:O gs://cbro-scratch

TestLogSync looks flaky, too. First run: logging_test.go:226: got 1 entries, want 2

Second run: logging_test.go:226: got 3 entries, want 2

On Wed, Aug 31, 2016 at 4:31 AM, Jonathan Amsterdam < notifications@github.com> wrote:

Chris,

First let's fix your local project.

Did you do

$ gsutil acl ch -u cloud-logs@system.gserviceaccount.com:O gs://cbro-scratch

On Wed, Aug 31, 2016 at 1:43 AM Chris Broadfoot notifications@github.com wrote:

Logging is better now, but still fails:

Sink: name:"GO-CLIENT-TEST-SINK-1472621764-0013" destination:"storage. googleapis.com/dulcet-port-762" filter:"logName = \"projects/dulcet-port-762/logs/GO-LOGGING-CLIENT-TEST-LOG\"" output_version_format:V2 --- FAIL: TestCreateDeleteSink (0.11s) sinks_test.go:72: rpc error: code = 7 desc = The caller does not have permission === RUN TestUpdateSink --- FAIL: TestUpdateSink (0.08s) sinks_test.go:107: rpc error: code = 7 desc = The caller does not have permission === RUN TestListSinks Sink: name:"GO-CLIENT-TEST-SINK-1472621764-0015" destination:"storage. googleapis.com/dulcet-port-762" filter:"logName = \"projects/dulcet-port-762/logs/GO-LOGGING-CLIENT-TEST-LOG\"" output_version_format:V2 --- FAIL: TestListSinks (0.08s) sinks_test.go:150: Create("GO-CLIENT-TEST-SINK-1472621764-0015"): rpc error: code = 7 desc = The caller does not have permission

@jba https://github.com/jba - that bucket has given the logs service account Owner permission. What's this failure? I reproduce this locally with my cbro-scratch project, too.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/gcloud-golang/issues/335# issuecomment-243664208, or mute the thread https://github.com/notifications/unsubscribe-auth/ARoHZWczrQeERxh_ pvI0fv1KpvDkWU7sks5qlRR2gaJpZM4Jw9Fr

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/gcloud-golang/issues/335#issuecomment-243736766, or mute the thread https://github.com/notifications/unsubscribe-auth/AABhlq6tnnGQ9lUMJbGfyPnm0POuih08ks5qlWYZgaJpZM4Jw9Fr .

broady commented 8 years ago

Nice work, everyone! We're down to two failures.

The permissions error for the log sink integration test (@jba and @broady investigating), and the change related to the Google API client (@shantuo - are you fixing this?).

broady commented 8 years ago

@jba and I initially missed one thing in the log export directions: the account performing the export must be an Owner or have the Logs Configuration Writer IAM role.

@jba added that to the service account used on Travis. I've kicked off another build to verify.

broady commented 8 years ago

Logging permissions work now, but the test fails with this error:

Sinks with different output formats cannot share destination. There already is a project sink with V1 output format.

jba commented 8 years ago

I've tried to fix that but can't. Asked for help from cloud logging folks.

On Wed, Aug 31, 2016 at 12:36 PM Chris Broadfoot notifications@github.com wrote:

Logging permissions work now, but the test fails with this error:

Sinks with different output formats cannot share destination. There already is a project sink with V1 output format.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/gcloud-golang/issues/335#issuecomment-243823197, or mute the thread https://github.com/notifications/unsubscribe-auth/ARoHZSMKfl4e_cY-k26c3MlTMydPIuRlks5qla1zgaJpZM4Jw9Fr .

broady commented 8 years ago

Yeah, not so great. How about creating a new bucket for each test?

jba commented 8 years ago

@okdave @cbro Are you okay with having logging tests depend on the storage client?

broady commented 8 years ago

Yep!

okdave commented 8 years ago

Yeah, that's fine.