search

weimeilin79 / cameleventmesh

Hybrid Multicloud Event Mesh with Camel
3 stars 3 forks source link

Gcpreader.java : Unable to mnt --configmap=gcp-configmap --configmap=kafka-config and PermissionDeniedException in K8s #1

Open qafro1 opened 3 years ago

qafro1 commented 3 years ago

Please I'm trying to install your GCP pubsub into k8s (minikube then GKE), however, I've got issues mounting the configs above resolved, but PermissionDeniedException is the problem. Issue: 

    ... 17 more
Caused by: io.grpc.StatusRuntimeException: PERMISSION_DENIED: The request is missing a valid API key.
    at io.grpc.Status.asRuntimeException(Status.java:533)
    ... 16 more

2021-07-26 10:46:24,031 ERROR [org.apa.cam.com.goo.pub.GooglePubsubConsumer] (Camel (camel-1) thread #0 - GooglePubsubConsumer[gcp-topic-sub]) Failure getting messages from PubSub: java.lang.IllegalStateException: Expected the service InnerService [FAILED] to be TERMINATED, but the service has FAILED
    at com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379)
    at com.google.common.util.concurrent.AbstractService.awaitTerminated(AbstractService.java:336)
    at com.google.api.core.AbstractApiService.awaitTerminated(AbstractApiService.java:104)
    at org.apache.camel.component.google.pubsub.GooglePubsubConsumer$SubscriberWrapper.asynchronousPull(GooglePubsubConsumer.java:133)
    at org.apache.camel.component.google.pubsub.GooglePubsubConsumer$SubscriberWrapper.run(GooglePubsubConsumer.java:115)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.google.api.gax.rpc.PermissionDeniedException: com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: The request is missing a valid API key.
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:55)
    at com.google.cloud.pubsub.v1.StreamingSubscriberConnection$1.onFailure(StreamingSubscriberConnection.java:267)
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1083)
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:760)
    at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95)
    at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77)
    at com.google.api.core.SettableApiFuture.setException(SettableApiFuture.java:52)
    at com.google.cloud.pubsub.v1.StreamingSubscriberConnection$StreamingPullResponseObserver.onError(StreamingSubscriberConnection.java:194)
    at com.google.api.gax.tracing.TracedResponseObserver.onError(TracedResponseObserver.java:103)
    at com.google.api.gax.grpc.ExceptionResponseObserver.onErrorImpl(ExceptionResponseObserver.java:84)
    at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:86)
    at com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onClose(GrpcDirectStreamController.java:149)
    at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:464)
    at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:428)
    at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:461)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:803)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:782)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    ... 3 more
Caused by: com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: The request is missing a valid API key.
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:55)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
    at com.google.api.gax.grpc.ExceptionResponseObserver.onErrorImpl(ExceptionResponseObserver.java:82)
    ... 17 more
Caused by: io.grpc.StatusRuntimeException: PERMISSION_DENIED: The request is missing a valid API key.
    at io.grpc.Status.asRuntimeException(Status.java:533)
    ... 16 more

2021-07-26 10:46:24,707 SEVERE [com.goo.clo.pub.v1.StreamingSubscriberConnection] (Subscriber-SE-362-0) terminated streaming with exception: com.google.api.gax.rpc.PermissionDeniedException: com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: The request is missing a valid API key.
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:55)

How do I modify the file #Kafka ConfigMap and #Google service key for k8s ?

#
# Quarkus
#
quarkus.banner.enabled = false

# Configure Quarkus Jaeger OpenTracing support
quarkus.jaeger.service-name = greetings-service
quarkus.jaeger.sampler-type = const
quarkus.jaeger.sampler-param = 1
quarkus.jaeger.endpoint = http://localhost:14268/api/traces

quarkus.kubernetes-client.trust-certs=true

# Camel
#
camel.context.name = camel-gcp

#kafka.brokers=my-cluster-kafka-bootstrap:9092
#kafka.brokers=localhost:9092

#Kafka ConfigMap
quarkus.openshift.env.configmaps=kafka-config

#Google service key
quarkus.openshift.config-map-volumes.gcp-volume.config-map-name=gcp-configmap
quarkus.openshift.mounts.gcp-volume.path=/mnt
quarkus.openshift.env.vars.GOOGLE_APPLICATION_CREDENTIALS=/mnt/google-service-acc-key.json

I'm checking out the doc and I've changed the quarkus.openshift. to quarkus.kubernetes . This has resolve the mnt issue and now is the problem . Any Ideas? Caused by: com.google.api.gax.rpc.PermissionDeniedException: com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: The request is missing a valid API key.

weimeilin79 commented 3 years ago

I can't replicate your problem,... Depends on the version of your K8s. I was using OpenShift 4.7. I guess you have to map it to what you are using? Maybe you want to try asking that in the Camel K community?

qafro1 commented 3 years ago

Please can you review my code gcp-kafka-sink because I just want to sent topics from Kafka into google pubsub. I think I've resolved the issue and this is the stream from my console but it does not stream to google pubsub. Please how to I sent the topic to google pubsub: topics/gcp-result

2021-07-26T16:28:31+01:00 2021-07-26 15:28:31,176 INFO  [route1] (Camel (camel-1) thread #0 - KafkaConsumer[gcp-result]) "Hello world - 271" Incident [] request for GCP resource update COMPLETED!!
2021-07-26T16:28:31+01:00 2021-07-26 15:28:31,277 INFO  [route1] (Camel (camel-1) thread #0 - KafkaConsumer[gcp-result]) "Hello world - 384" Incident [] request for GCP resource update COMPLETED!!
2021-07-26T16:28:31+01:00 2021-07-26 15:28:31,281 INFO  [route1] (Camel (camel-1) thread #0 - KafkaConsumer[gcp-result]) "Hello world - 588" Incident [] request for GCP resource update COMPLETED!!
2021-07-26T16:28:31+01:00 2021-07-26 15:28:31,288 INFO  [route1] (Camel (camel-1) thread #0 - KafkaConsumer[gcp-result]) "Hello world - 373" Incident [] request for GCP resource update COMPLETED!!
2021-07-26T16:28:31+01:00 2021-07-26 15:28:31,297 INFO  [route1] (Camel (camel-1) thread #0 - KafkaConsumer[gcp-result]) "Hello world - 654" Incident [] request for GCP resource update COMPLETED!!
2021-07-26T16:28:31+01:00 2021-07-26 15:28:31,377 INFO  [route1] (Camel (camel-1) thread #0 - KafkaConsumer[gcp-result]) "Hello world - 581" Incident [] request for GCP resource update COMPLETED!!
2021-07-26T16:28:31+01:00 2021-07-26 15:28:31,472 INFO  [route1] (Camel (camel-1) thread #0 - KafkaConsumer[gcp-result]) "Hello world - 647" Incident [] request for GCP resource update COMPLETED!!
2021-07-26T16:28:31+01:00 2021-07-26 15:28:31,475 INFO  [route1] (Camel (camel-1) thread #0 - KafkaConsumer[gcp-result]) "Hello world - 399" Incident [] request for GCP resource update COMPLETED!!
2021-07-26T16:28:31+01:00 2021-07-26 15:28:31,544 INFO  [route1] (Camel (camel-1) thread #0 - KafkaConsumer[gcp-result]) "Hello world - 19" Incident [] request for GCP resource update COMPLETED!!
2021-07-26T16:28:31+01:00 2021-07-26 15:28:31,549 INFO  [route1] (Camel (camel-1) thread #0 - KafkaConsumer[gcp-result]) "Hello world - 763" Incident [] request for GCP resource update COMPLETED!!
2021-07-26T16:28:31+01:00 2021-07-26 15:28:31,563 INFO  [route1] (Camel (camel-1) thread #0 - KafkaConsumer[gcp-result]) "Hello world - 768" Incident [] request for GCP resource update COMPLETED!!