GCP PubSub Authorisation Issue In Java Using Service Account

shubhwip commented 5 months ago

Describe the bug I am facing below issue while publishing messages to various topics from one java service.

exception is com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.\ncom.google.cloud.spring.pubsub.core.PubSubDeliveryException: Publishing to projects/testing-development/topics/testing-bacd-testing-1 topic failed.; nested exception is com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.\n\tat com.google.cloud.spring.pubsub.core.publisher.PubSubPublisherTemplate$1.onFailure(PubSubPublisherTemplate.java:111)\n\tat com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)\n\tat com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1074)\n\tat com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)\n\tat com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)\n\tat com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)\n\tat com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)\n\tat com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95)\n\tat com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77)\n\tat com.google.api.core.SettableApiFuture.setException(SettableApiFuture.java:52)\n\tat com.google.cloud.pubsub.v1.Publisher$OutstandingBatch.onFailure(Publisher.java:545)\n\tat com.google.cloud.pubsub.v1.Publisher$OutstandingBatch.access$1600(Publisher.java:512)\n\tat com.google.cloud.pubsub.v1.Publisher$3.onFailure(Publisher.java:488)\n\tat com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)\n\tat com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1074)\n\tat com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)\n\tat com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)\n\tat com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)\n\tat com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)\n\tat com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:200)\n\tat com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135)\n\tat com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:117)\n\tat com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)\n\tat com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)\n\tat com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)\n\tat com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)\n\tat com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95)\n\tat com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77)\n\tat com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)\n\tat com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)\n\tat com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1074)\n\tat com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)\n\tat com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)\n\tat com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)\n\tat com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)\n\tat io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:563)\n\tat io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460)\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:553)\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:68)\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:739)\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:718)\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.\n\tat com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:55)\n\tat com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)\n\tat com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)\n\t... 24 common frames omitted\nCaused by: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.\n\tat io.grpc.Status.asRuntimeException(Status.java:535)\n\t... 16 common frames omitted\n","context":"default"}

My Setup

We are trying to publish to many topics from one java service. We have created

topics1 => using a wrapper library over spring-cloud-gcp-pubsub and using it to publish to some of the topics topics2 => using spring-cloud-gcp-pubsub directly to publish to other topics and this is behind a conditional property. We are seeing above error for all the topics in topics1 when conditional property for topics2 is enabled.

Attempts

There was library version mismatch, fixed it.
Tried to run it locally with minimal setup and it is working fine.
Checked permissions of service account being used and it has all the required permissions.
Enabled pubsub auditing as well to find out but pubsub doesn't log Publish API calls.

I would like to find out what is this user who is not authorised. Is there some access_token invalidation happening ?

Thanks in advance.

burkedavison commented 5 months ago

I would like to find out what is this user who is not authorised. Is there some access_token invalidation happening ?

You can enable logging for the auth library to see if it provides any additional clues as to what is happening -- see the logging section of the Google java client library TROUBLESHOOTING.md for an example of how to do that.

shubhwip commented 5 months ago

Thanks @burkedavison, I have a bearer token with me, can i decode this by any chance ? I am suspecting if scopes are getting changed.

{"timestampSeconds":1712307807,"timestampNanos":830000000,"severity":"INFO","thread":"Gax-14","logger":"com.google.api.client.http.HttpTransport","message":"{\"access_token\":\"XXXXXXXXXXXX\",\"expires_in\":3598,\"token_type\":\"Bearer\"}","context":"default"}

Publish Request

{"timestampSeconds":1712307913,"timestampNanos":796000000,"severity":"DEBUG","thread":"grpc-default-worker-ELG-1-1","logger":"io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler","message":"[id: 0xe0a011e9, L:/10.70.5.54:37214 - R:pubsub.googleapis.com/108.177.15.95:443] OUTBOUND HEADERS: streamId\u003d9 headers\u003dGrpcHttp2OutboundHeaders[:authority: pubsub.googleapis.com:443, :path: /google.pubsub.v1.Publisher/Publish, :method: POST, :scheme: https, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.39.0, x-goog-api-client: gl-java/17.0.4 gapic/1.113.5 gax/1.66.0 grpc/1.39.0, x-goog-request-params: topic\u003dprojects/testing-development/topics/testing-bacd-testing-1, grpc-accept-encoding: gzip, authorization: Bearer XXXXXXXXXXXX, grpc-timeout: 4999700u] streamDependency\u003d0 weight\u003d16 exclusive\u003dfalse padding\u003d0 endStream\u003dfalse","context":"default"}

I tried this

curl -i "https://oauth2.googleapis.com/tokeninfo?access_token=$TOKEN"
HTTP/2 400 
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: Mon, 01 Jan 1990 00:00:00 GMT
date: Fri, 05 Apr 2024 09:06:40 GMT
content-type: application/json; charset=UTF-8
vary: X-Origin
vary: Referer
vary: Origin,Accept-Encoding
server: ESF
x-xss-protection: 0
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
accept-ranges: none

{
  "error": "invalid_token",
  "error_description": "Invalid Value"
}

meltsufin commented 5 months ago

@westarle Can you please help with the question about decoding Bearer tokens?

westarle commented 5 months ago

@shubhwip Does the token look like ya29.... or does it look like a JWT-structured token? If it is the latter, you can use something like jwt.io to decode!

burkedavison commented 2 months ago

Closing due to inactivity.

GoogleCloudPlatform / spring-cloud-gcp