GRPC Plugin

[Null1515]

Hey there,

First of all, I want to thank you for this amazing piece of art! Your work has been incredibly valuable.

I noticed that there is a gRPC plugin available in the source code, and I’m interested in utilizing it. However, I’m not sure how to go about using it effectively.

Could you please provide some guidance or documentation on how to use the gRPC plugin? Any tips or examples would be greatly appreciated!

Thank you for your help!

[frankfliu]

@Null1515

Currently gRPC plugin is not installed by default, here is how to install this plugin:

1. clone djl-serving: git clone https://github.com/deepjavalibrary/djl.git
2. build grpc plugin: cd plugins && ./gradlew jar
3. copy plugin jar file into your djl-serving installation folder: cp build/libs/grpc-0.31.0-SNAPSHOT.jar /usr/local/djl-serving-0.31.0-SNAPSHOT/plugins/
4. start djl-serving: djl-serving, grpc service will be listening on 8082 port

You can use inference.proto to create your own grpc client to run inference against djl-serving. See example code: https://github.com/deepjavalibrary/djl-serving/blob/master/plugins/grpc/src/test/java/ai/djl/serving/grpc/GrpcTest.java#L69

[Null1515]

Hello again,

Thank you for your previous response—I really appreciate your help!

I wanted to follow up on the earlier discussion. I have attempted the steps outlined, but it seems that the gRPC plugin isn't being added successfully. I suspect this may be due to a missing component in its JAR file. Additionally, I plan to use this within a Docker environment, which adds another layer of complexity.

I also have another request regarding inference from a Java client using a BufferedImage. The current implementation requires the use of the ImageIO package to encode the image into a byte array. However, this encoding process can be quite time-consuming. For instance, encoding an image at a resolution of 1920x1080 takes approximately 200ms for PNG and 40ms for BMP formats.

Could you please provide guidance on how to utilize a direct byte array from a BufferedImage with your inference API? Below is the code snippet I’m currently working with:

// Image to Direct Byte Array
WritableRaster raster = image.getRaster();
DataBufferByte data = (DataBufferByte) raster.getDataBuffer();
byte[] dataBuffer = data.getData();

// Direct Byte Array to Image
DataBuffer buffer = new DataBufferByte(dataBuffer, dataBuffer.length);

BufferedImage image1;
boolean hasAlpha = image.getColorModel().hasAlpha();
if (hasAlpha) {
    ColorModel cm = new ComponentColorModel(ColorSpace.getInstance(ColorSpace.CS_sRGB), new int[]{8, 8, 8, 8}, true,
            false, Transparency.TRANSLUCENT, DataBuffer.TYPE_BYTE);
    image1 = new BufferedImage(cm, Raster.createInterleavedRaster(buffer, imageWidth, imageHeight, imageWidth * 4, 4, new int[]{2, 1, 0, 3}, null),
            false, null);
} else {
    ColorModel cm = new ComponentColorModel(ColorSpace.getInstance(ColorSpace.CS_sRGB), new int[]{8, 8, 8, 8}, false,
            false, 1, DataBuffer.TYPE_BYTE);
    image1 = new BufferedImage(cm, Raster.createInterleavedRaster(buffer, imageWidth, imageHeight, imageWidth * 3, 3, new int[]{2, 1, 0}, null),
            false, null);
}

Thank you for your assistance!

[frankfliu]

@Null1515

1. What error message are you seeing? did you see something like: INFO GrpcServerImpl gRPC bind to port: 127.0.0.1:8082?
2. build docker image should be pretty straightforward. you can use djl-serving docker as your base image, docker file can be found: https://github.com/deepjavalibrary/djl-serving/tree/master/serving/docker, and the image is published to dockerhub: https://hub.docker.com/r/deepjavalibrary/djl-serving/tags
3. For image process, I recommend you to use ImageFactory, if the performance is critical, you can use OpenCVImageFactory

[Null1515]

Hello,

I would like to bring to your attention some details from the log file regarding the plugin initialization process:

INFO FolderScanPluginManager scanning for plugins... INFO FolderScanPluginManager scanning in plug-in folder: /usr/local/djl-serving-0.28.0/plugins INFO PropertyFilePluginMetaDataReader Plugin found: console/jar:file:/usr/local/djl-serving-0.28.0/plugins/management-console-0.28.0.jar!/META-INF/plugin.definition INFO PropertyFilePluginMetaDataReader Plugin found: cache-engines/jar:file:/usr/local/djl-serving-0.28.0/plugins/cache-0.28.0.jar!/META-INF/plugin.definition INFO PropertyFilePluginMetaDataReader Plugin found: static-file-plugin/jar:file:/usr/local/djl-serving-0.28.0/plugins/static-file-plugin-0.28.0.jar!/META-INF/plugin.definition INFO PropertyFilePluginMetaDataReader Plugin found: secure-mode/jar:file:/usr/local/djl-serving-0.28.0/plugins/secure-mode-0.28.0.jar!/META-INF/plugin.definition INFO PropertyFilePluginMetaDataReader Plugin found: kserve/jar:file:/usr/local/djl-serving-0.28.0/plugins/kserve-0.28.0.jar!/META-INF/plugin.definition INFO FolderScanPluginManager Loading plugin: {console/jar:file:/usr/local/djl-serving-0.28.0/plugins/management-console-0.28.0.jar!/META-INF/plugin.definition} INFO PluginMetaData plugin console changed state to INITIALIZED INFO FolderScanPluginManager Loading plugin: {static-file-plugin/jar:file:/usr/local/djl-serving-0.28.0/plugins/static-file-plugin-0.28.0.jar!/META-INF/plugin.definition} INFO PluginMetaData plugin static-file-plugin changed state to INITIALIZED INFO FolderScanPluginManager Loading plugin: {cache-engines/jar:file:/usr/local/djl-serving-0.28.0/plugins/cache-0.28.0.jar!/META-INF/plugin.definition} INFO PluginMetaData plugin cache-engines changed state to INITIALIZED INFO FolderScanPluginManager Loading plugin: {secure-mode/jar:file:/usr/local/djl-serving-0.28.0/plugins/secure-mode-0.28.0.jar!/META-INF/plugin.definition} INFO PluginMetaData plugin secure-mode changed state to INITIALIZED INFO FolderScanPluginManager Loading plugin: {kserve/jar:file:/usr/local/djl-serving-0.28.0/plugins/kserve-0.28.0.jar!/META-INF/plugin.definition} INFO PluginMetaData plugin kserve changed state to INITIALIZED INFO PluginMetaData plugin console changed state to ACTIVE reason: plugin ready INFO PluginMetaData plugin static-file-plugin changed state to ACTIVE reason: plugin ready INFO PluginMetaData plugin cache-engines changed state to ACTIVE reason: plugin ready INFO PluginMetaData plugin secure-mode changed state to ACTIVE reason: plugin ready INFO PluginMetaData plugin kserve changed state to ACTIVE reason: plugin ready

While the plugins have been successfully initialized, I want to note that the gRPC plugin is not being added. I will attach the relevant .jar plugin files for your reference.

Additionally, I have a couple of questions:

How can I add a gRPC plugin to the Docker image? I am currently using FFmpeg along with other packages and would like to utilize BufferedImage as a source. Can you confirm if this is a viable method to send my BufferedImage to djl-serve? [Uploading grpc.zip…]()

[frankfliu]

@Null1515 You are using 0.28.0, which not supported. You need to use 0.31.0-SNAPSHOT

[Null1515]

Thanks for your last answer! I wanted to follow up on my previous question about utilizing BufferedImage as a source. I am currently using FFmpeg along with other packages and would like to know if you can confirm if using OpenCV extension is a viable method to send my BufferedImage to djl-serve. If you have any updates or insights, I’d really appreciate it!

[frankfliu]

@Null1515

I'm not sure about that. The OpenCV extension we use (depends on org.openpnp:opencv) might have conflict with FFmpeg. For video use case, I think the best way is to create a video extension similar to https://github.com/deepjavalibrary/djl/tree/master/extensions/audio

And implement ImageFactory based on org.bytedeco:opencv package.

[Null1515]

How can i implement my own byte array decoder in djl serve?

[frankfliu]

I assume you are using org.bytedeco:ffmpeg, If org.openpnp:opencv not conflict with org.bytedeco:ffmpeg, you can directly copy OpenCVImageFactory directly. You can find implementation for converting Mat to NDArray here: https://github.com/deepjavalibrary/djl/blob/master/extensions/opencv/src/main/java/ai/djl/opencv/OpenCVImage.java#L134

You can implement your own Translator in Java use any dependencies your want to use. You can package your java code with your model, see: https://docs.djl.ai/master/docs/create_serving_ready_model.html#bundle-your-data-processing-scripts-together-with-model-artifacts

After package your model, you also need copy all your 3rd party dependencies in deps folder in djl-serving:

djl-serving -i org.openpnp:opencv:4.9.0-1
djl-serving -i ai.djl.opencv:opencv:0.30.0
...