search

spring-attic / spring-cloud-gcp

Integration for Google Cloud Platform APIs with Spring
Apache License 2.0
705 stars 693 forks source link

Bad credentials file causes Pub/Sub topic creation to hang #962

Open meltsufin opened 6 years ago

meltsufin commented 6 years ago

Steps to reproduce:

  1. Create and download a GCP service account key (JSON file).
  2. Make sure the key is actually deleted in the Cloud Console so that it's invalid.
  3. Modify application.properties in spring-cloud-gcp-pubsub-sample to have spring.cloud.gcp.credentials.location point to the download key file.
  4. Run the application and try to create a new topic using the web UI.

Observe that the application just hangs with no error messages.

jlheard commented 6 years ago

This repeats over and over in the logs when running an Integration test after the key has been deleted:

15:28:04.944 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] HANDSHAKEN: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
15:28:04.944 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] OUTBOUND SETTINGS: ack=false settings={ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=0, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=32768}
15:28:04.944 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041
15:28:04.946 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] INBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=16384}
15:28:04.946 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] OUTBOUND SETTINGS: ack=true
15:28:04.946 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041
15:28:05.011 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] INBOUND SETTINGS: ack=true
15:32:04.945 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] INBOUND GO_AWAY: lastStreamId=0 errorCode=0 length=17 bytes=73657373696f6e5f74696d65645f6f7574
meltsufin commented 6 years ago

Thanks @jlheard! We're waiting for the fix in the client library. See: https://github.com/GoogleCloudPlatform/google-cloud-java/issues/3573.

dzou commented 5 years ago

Just tried running this; this appears to still be an issue.

meltsufin commented 5 years ago

They still need to fix it in https://github.com/googleapis/google-cloud-java/issues/3573.

elefeint commented 4 years ago

Underlying issue seems to have been fixed?

meltsufin commented 4 years ago

It looks like it's fixed, but we should probably re-run the test to confirm before closing.

dzou commented 4 years ago

Hmm I don't think they ended up changing anything. The recommendation they provided in the comment https://github.com/googleapis/google-cloud-java/issues/3573#issuecomment-578985763 suggested that we could modify our RPC timeout setting for our topicAdminClient to 30 seconds instead of 10 minutes tho which would eventually allow it to return if the credentials file is invalid.

Is this something we should modify in our autoconfiguration?

meltsufin commented 4 years ago

Do we expose this as a configuration property?

dzou commented 4 years ago

No, but it is a bean the user can override: https://github.com/spring-cloud/spring-cloud-gcp/blob/master/spring-cloud-gcp-autoconfigure/src/main/java/org/springframework/cloud/gcp/autoconfigure/pubsub/GcpPubSubAutoConfiguration.java#L351

Still tho it doesn't feel quite right to ask the user to override the setting just so they'll time out quicker if they accidentally use an incorrect auth config since the user will most likely encounter the issue before they learn about the setting.

dzou commented 4 years ago

Let me dig into the client library code, see if there is something we could change to deal with this better. Otherwise I would recommend maybe setting the timeout from 10 mins to 1 minute for topic admin?

elefeint commented 4 years ago

Can we pre-check if the key is valid? Or will it require resourcemanager permissions or something...

dzou commented 4 years ago

Posted question in GRPC; hopefully can get a reaction there: https://github.com/grpc/grpc-java/issues/6808