search

triton-inference-server / server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.
https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
BSD 3-Clause "New" or "Revised" License
8.31k stars 1.48k forks source link

Facing issue with setting up GRPC client in kotlin for using Gradle #6597

Open AyushVachaspati opened 12 months ago

AyushVachaspati commented 12 months ago

Hi, This is probably not the best place to ask this, but since this community is probably more familiar with setting up GRPC client side code for Trition than the general internet, I'm trying my luck.

I'm trying to setup grpc client to make inference requests to Triton (version:23.06-py3) in Kotlin for my project.

I've setup protoc code generation using gradle (attached build.gradle.kts below), but when I try to create a ManagedChannel to pass to the InferenceService Coroutine, I get this Runtime Error

val channel = ManagedChannelBuilder.forTarget("127.0.0.1:8001")
            .usePlainText()
            .build()
val inferenceStub = GRPCInferenceServiceGrpcKt.GRPCInferenceServiceCoroutineStub(channel)
val modelLive = inferenceStub.modelReady(
    GrpcService.ModelReadyRequest.newBuilder()
         .setName("santacoder_huggingface")
         .build()
)
2023-11-17 15:11:36,839 [   9708]  ERROR - pplication.impl.LaterInvocator - loader constraint violation: loader com.intellij.util.lang.UrlClassLoader @174d20a wants to load abstract class io.grpc.Channel. A different abstract class with the same name was previously loaded by com.intellij.ide.plugins.cl.PluginClassLoader @6404ea6d. (io.grpc.Channel is in unnamed module of loader com.intellij.ide.plugins.cl.PluginClassLoader @6404ea6d, parent loader 'bootstrap') 
java.lang.LinkageError: loader constraint violation: loader com.intellij.util.lang.UrlClassLoader @174d20a wants to load abstract class io.grpc.Channel. A different abstract class with the same name was previously loaded by com.intellij.ide.plugins.cl.PluginClassLoader @6404ea6d. (io.grpc.Channel is in unnamed module of loader com.intellij.ide.plugins.cl.PluginClassLoader @6404ea6d, parent loader 'bootstrap')
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:878)
    at com.intellij.util.lang.UrlClassLoader._defineClass(UrlClassLoader.java:400)
    at com.intellij.util.lang.UrlClassLoader.defineClass(UrlClassLoader.java:392)
    at com.intellij.util.lang.UrlClassLoader._findClass(UrlClassLoader.java:350)
    at com.intellij.util.lang.UrlClassLoader.findClass(UrlClassLoader.java:336)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at io.grpc.stub.AbstractStub.<init>(AbstractStub.java:73)
    at io.grpc.kotlin.AbstractCoroutineStub.<init>(AbstractCoroutineStub.kt:34)
    at inference.GRPCInferenceServiceGrpcKt$GRPCInferenceServiceCoroutineStub.<init>(GrpcServiceGrpcKt.kt:146)
    at inference.GRPCInferenceServiceGrpcKt$GRPCInferenceServiceCoroutineStub.<init>(GrpcServiceGrpcKt.kt:143)
    at com.artemus.completionProvider.ArtemusCompletionProvider.getInlineCompletion(ArtemusCompletionProvider.kt:16)
    at com.artemus.inlineCompletionApi.InlineCompletionsManager$createPreviewInline$1.invokeSuspend(InlineCompletionsManager.kt:27)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
    at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:284)
    at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
    at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
    at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
    at com.artemus.inlineCompletionApi.InlineCompletionsManager.createPreviewInline(InlineCompletionsManager.kt:27)
    at com.artemus.inlineCompletionApi.listeners.DocumentListener.documentChangedNonBulk$lambda$2(DocumentListener.kt:78)
    at com.intellij.openapi.application.TransactionGuardImpl.runWithWritingAllowed(TransactionGuardImpl.java:216)
    at com.intellij.openapi.application.TransactionGuardImpl.access$200(TransactionGuardImpl.java:24)
    at com.intellij.openapi.application.TransactionGuardImpl$2.run(TransactionGuardImpl.java:199)
    at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:822)
    at com.intellij.openapi.application.impl.ApplicationImpl.lambda$invokeLater$4(ApplicationImpl.java:324)
    at com.intellij.openapi.application.impl.FlushQueue.doRun(FlushQueue.java:85)
    at com.intellij.openapi.application.impl.FlushQueue.runNextEvent(FlushQueue.java:134)
    at com.intellij.openapi.application.impl.FlushQueue.flushNow(FlushQueue.java:47)
    at com.intellij.openapi.application.impl.FlushQueue$FlushNow.run(FlushQueue.java:190)
    at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:313)
    at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:776)
    at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:727)
    at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
    at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:746)
    at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:976)
    at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:843)
    at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$8(IdeEventQueue.java:454)
    at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:773)
    at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$9(IdeEventQueue.java:453)
    at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:822)
    at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:501)
    at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
    at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
    at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
    at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
    at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
    at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)
2023-11-17 15:11:36,841 [   9710]  ERROR - pplication.impl.LaterInvocator - IntelliJ IDEA 2020.3  Build #IC-203.5981.155 
2023-11-17 15:11:36,842 [   9711]  ERROR - pplication.impl.LaterInvocator - JDK: 11.0.9; VM: OpenJDK 64-Bit Server VM; Vendor: JetBrains s.r.o. 
2023-11-17 15:11:36,842 [   9711]  ERROR - pplication.impl.LaterInvocator - OS: Windows 10 
2023-11-17 15:11:36,842 [   9711]  ERROR - pplication.impl.LaterInvocator - Plugin to blame: Artemus version: 0.0.1 
2023-11-17 15:11:36,842 [   9711]  ERROR - pplication.impl.LaterInvocator - Last Action: EditorEnter 
fun properties(key: String) = providers.gradleProperty(key)
fun environment(key: String) = providers.environmentVariable(key)

plugins {
  id("java")
  alias(libs.plugins.kotlin) // Kotlin support
  alias(libs.plugins.gradleIntelliJPlugin) // Gradle IntelliJ Plugin
  alias(libs.plugins.changelog) // Gradle Changelog Plugin
  alias(libs.plugins.qodana) // Gradle Qodana Plugin
  id("com.google.protobuf") version "0.9.4"
}

group = properties("pluginGroup").get()
version = properties("pluginVersion").get()

repositories {
  mavenCentral()
}

kotlin {
  jvmToolchain(11)
}

intellij {
  pluginName = properties("pluginName")
  version = properties("platformVersion")
  type = properties("platformType")

  plugins = properties("platformPlugins").map { it.split(',').map(String::trim).filter(String::isNotEmpty) }
}

dependencies {
  implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.3")
  implementation("org.jetbrains.kotlinx:kotlinx-coroutines-swing:1.6.3")

  implementation("io.grpc:grpc-netty-shaded:1.59.0")
  implementation("io.grpc:grpc-protobuf:1.59.0")
  implementation("io.grpc:grpc-stub:1.59.0")

  implementation("com.google.protobuf:protobuf-java:3.25.1")
  implementation("com.google.protobuf:protobuf-java-util:3.25.1")
  implementation("com.google.protobuf:protobuf-kotlin:3.25.1")
  implementation("io.grpc:protoc-gen-grpc-kotlin:1.4.0")

}

protobuf {
  protoc {
    artifact = "com.google.protobuf:protoc:3.25.1"
  }

  plugins {
    create("grpc") {
      artifact = "io.grpc:protoc-gen-grpc-java:1.59.0"
    }
    create("grpckt") {
      artifact = "io.grpc:protoc-gen-grpc-kotlin:1.4.0:jdk8@jar"
    }
  }
  generateProtoTasks {
    all().forEach {
      it.plugins {
        create("grpc")
        create("grpckt")
      }
      it.builtins {
        create("kotlin")
      }
    }
  }
}

tasks {
  wrapper {
    gradleVersion = properties("gradleVersion").get()
  }

  patchPluginXml {
    version = properties("pluginVersion")
    sinceBuild = properties("pluginSinceBuild")
    untilBuild = properties("pluginUntilBuild")
  }

  signPlugin {
    certificateChain = environment("CERTIFICATE_CHAIN")
    privateKey = environment("PRIVATE_KEY")
    password = environment("PRIVATE_KEY_PASSWORD")
  }

  publishPlugin {
    token = environment("PUBLISH_TOKEN")
    channels = properties("pluginVersion").map { listOf(it.split('-').getOrElse(1) { "default" }.split('.').first()) }
  }
}

I thought this might be caused by grpc-netty-shaded dependency I have in the gradle file but I'm not sure since I'm very new to this. I tried removing grpc-netty-shaded in favour of grpc-netty, and also tried to remove the dependency entiry, but both of them lead to this error

2023-11-17 15:25:48,470 [   7696]  ERROR - pplication.impl.LaterInvocator - io.grpc.ManagedChannelProvider: io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider not a subtype 
java.util.ServiceConfigurationError: io.grpc.ManagedChannelProvider: io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider not a subtype
    at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:588)
    at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1236)
    at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1264)
    at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1299)
    at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1384)
    at io.grpc.ServiceProviders.loadAll(ServiceProviders.java:67)
    at io.grpc.ManagedChannelRegistry.getDefaultRegistry(ManagedChannelRegistry.java:101)
    at io.grpc.ManagedChannelProvider.provider(ManagedChannelProvider.java:43)
    at io.grpc.ManagedChannelBuilder.forTarget(ManagedChannelBuilder.java:86)
    at com.artemus.completionProvider.ArtemusCompletionProvider.getInlineCompletion(ArtemusCompletionProvider.kt:13)
    at com.artemus.inlineCompletionApi.InlineCompletionsManager$createPreviewInline$1.invokeSuspend(InlineCompletionsManager.kt:27)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
    at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:284)
    at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
    at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
    at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
    at com.artemus.inlineCompletionApi.InlineCompletionsManager.createPreviewInline(InlineCompletionsManager.kt:27)
    at com.artemus.inlineCompletionApi.listeners.DocumentListener.documentChangedNonBulk$lambda$2(DocumentListener.kt:78)
    at com.intellij.openapi.application.TransactionGuardImpl.runWithWritingAllowed(TransactionGuardImpl.java:216)
    at com.intellij.openapi.application.TransactionGuardImpl.access$200(TransactionGuardImpl.java:24)
    at com.intellij.openapi.application.TransactionGuardImpl$2.run(TransactionGuardImpl.java:199)
    at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:822)
    at com.intellij.openapi.application.impl.ApplicationImpl.lambda$invokeLater$4(ApplicationImpl.java:324)
    at com.intellij.openapi.application.impl.FlushQueue.doRun(FlushQueue.java:85)
    at com.intellij.openapi.application.impl.FlushQueue.runNextEvent(FlushQueue.java:134)
    at com.intellij.openapi.application.impl.FlushQueue.flushNow(FlushQueue.java:47)
    at com.intellij.openapi.application.impl.FlushQueue$FlushNow.run(FlushQueue.java:190)
    at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:313)
    at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:776)
    at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:727)
    at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
    at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:746)
    at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:976)
    at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:843)
    at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$8(IdeEventQueue.java:454)
    at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:773)
    at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$9(IdeEventQueue.java:453)
    at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:822)
    at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:501)
    at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
    at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
    at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
    at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
    at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
    at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)
2023-11-17 15:25:48,472 [   7698]  ERROR - pplication.impl.LaterInvocator - IntelliJ IDEA 2020.3  Build #IC-203.5981.155 
2023-11-17 15:25:48,473 [   7699]  ERROR - pplication.impl.LaterInvocator - JDK: 11.0.9; VM: OpenJDK 64-Bit Server VM; Vendor: JetBrains s.r.o. 
2023-11-17 15:25:48,473 [   7699]  ERROR - pplication.impl.LaterInvocator - OS: Windows 10 
2023-11-17 15:25:48,473 [   7699]  ERROR - pplication.impl.LaterInvocator - Plugin to blame: Artemus version: 0.0.1 
2023-11-17 15:25:48,473 [   7699]  ERROR - pplication.impl.LaterInvocator - Last Action: EditorEnter

I hope someone here is able to point out the, probably obvious, mistake I'm making. I'm really new to Kotlin, Gradle and all of this, and I'd really appreciate any I help I can get.

Thanks! Ayush

kthui commented 11 months ago

Hi @AyushVachaspati, we don't officially support Kotlin, but the underlying gRPC connection should be the same. I think the Triton Java API example might help: https://github.com/triton-inference-server/client/tree/main/src/java

AyushVachaspati commented 11 months ago

Hi @kthui , thanks for responding. It seems that the issue is with Class Loaders for IntelliJ and netty using incompatible classLoaders or something. I've raised the issue with JetBrains and with grpc-java. Maybe they'll be able to help.

This might not have been the best place to ask this, but I still asked hoping that someone in this community would have maybe tried creating an IntelliJ plugin using grpc before and encountered/solved the issue.

kthui commented 11 months ago

Maybe @nnshah1 might have some resources on this issue?