Closed shril closed 10 months ago
You need to make sure that you are updating the service files when repackaging grpc. In Maven you would use something like: https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
However I'm uncertain whats the equivalent for sbt, but the resulting shaded jar needs to have a META-INF/services files that are correctly updated to with the repackaged class names
@igorbernstein2 I did the following and it worked perfectly well for me.
In src/main/resources
I added META-INF/services
folder.
In the services
folder I added 2 files namely
io.grpc.LoadBalancerProvider
io.grpc.NameResolverProvider
The content of both the files are as follows -
io.grpc.LoadBalancerProvider
io.grpc.internal.PickFirstLoadBalancerProvider
io.grpc.util.SecretRoundRobinLoadBalancerProvider$Provider
io.grpc.util.OutlierDetectionLoadBalancerProvider
and io.grpc.NameResolverProvider
io.grpc.internal.DnsNameResolverProvider
I am trying to run a Spark Application to write and read data to Cloud Bigtable from Dataproc.
Initially, I got this exception
java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument
. Then came to know that there are some dependency issues from this Google Documentation - Manage Java and Scala dependencies for Apache Spark.Following the instructions, I changed my
build.sbt
file to shade the jars -Then got this error
Following that, I added the dependency of in my
build.sbt
file.Still I am getting the same error.
Environment details
Dataproc details -
Spark Job details -
Can provide any additional details if required? Thanks!