Error reported on incoming file using djl-serving on Windows

[Bill-zy]

I used the inference API to pass in a file and the following error occurred：

Caused by: java.lang.IllegalArgumentException: Malformed data at ai.djl.ndarray.NDList.decode(NDList.java:124) ~[api-0.19.0.jar:?] at ai.djl.ndarray.NDList.decode(NDList.java:85) ~[api-0.19.0.jar:?] at ai.djl.modality.Input.getAsNDList(Input.java:328) ~[api-0.19.0.jar:?] at ai.djl.modality.Input.getDataAsNDList(Input.java:198) ~[api-0.19.0.jar:?] at ai.djl.translate.NoopServingTranslatorFactory$NoopServingTranslator.processInput(NoopServingTranslatorFactory.java:68) ~[api-0.19.0.jar:?] ... 8 more Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:202) ~[?:?] at java.io.DataInputStream.readFully(DataInputStream.java:170) ~[?:?] at ai.djl.ndarray.NDList.decode(NDList.java:99) ~[api-0.19.0.jar:?] at ai.djl.ndarray.NDList.decode(NDList.java:85) ~[api-0.19.0.jar:?] at ai.djl.modality.Input.getAsNDList(Input.java:328) ~[api-0.19.0.jar:?] at ai.djl.modality.Input.getDataAsNDList(Input.java:198) ~[api-0.19.0.jar:?] at ai.djl.translate.NoopServingTranslatorFactory$NoopServingTranslator.processInput(NoopServingTranslatorFactory.java:68) ~[api-0.19.0.jar:?]

[frankfliu]

@Bill-zy It looks like you didn't configure any translator in your model package. In this case, DJLServing will assume your input file is NDList encoded (or .npz numpy file).

Can you share which model you are using and which input are you sending to the server?

[Bill-zy]

I input a file path to the server. I followed the java-demo instructions to put the custom translator in the /libs/classes folder. But a custom translator doesn't seem to be in use. One question I have is how should I specify a custom translator when using the api to access it

[Bill-zy]

I put a "filePath" parameter in the request header. The code is as follows: public NDList processInput(TranslatorContext ctx, Input input) throws Exception { String filePath = input.getProperties().get("filePath"); ImageFactory factory = ImageFactory.getInstance(); Image image = factory.fromUrl(filePath); NDArray ndArray = image.toNDArray(ctx.getNDManager()); Pipeline pipeline = new Pipeline(); pipeline.add(new Resize(320)) .add(new ToTensor()) .add(new Normalize(new float[]{0.4706145f,0.46000465f,0.45479808f},new float[]{0.26668432f,0.26578658f,0.2706199f})); return pipeline.transform(new NDList(ndArray)); }

[frankfliu]

The easiest way to specify translator is:

1. create a TranslatorFactoryFactory class and create your Translator with the Factory, in this way you can pass whatever parameters in your Translator's constructor`
2. add serving.properties file in the model directory, and add translatorFacotry with full qualified class name, for example:

   translatorFactory=ai.djl.modality.cv.translator.ImageClassificationTranslatorFactory

3. compile your class and package into a jar file (with this way, you can find issue locally and easy to debug)
4. drop the jar file into <MODEL_SERVER_HOME>/lib/ folder

[Bill-zy]

linux: windows: Des: Starting the server in linux will look for translator in the /libs/classes/ folder. But windows doesn't look for it.What's the reason for this?

[frankfliu]

@Bill-zy Can you create a github repo to reproduce this issue, or simply share your model .zip file?

I'm not able to reproduce this issue. Our unittest seems working on Windows: https://github.com/deepjavalibrary/djl/blob/master/integration/src/main/java/ai/djl/integration/tests/model_zoo/CustomTranslatorTest.java#L145

Here is a sample model with custom Translator: mlp.zip, it's working fine on Windows:

[INFO ] ModelServer - Initializing model: mlp=file:///c:/source/models/mlp
[INFO ] WorkerPool - initWorkers for mlp (cpu()): -1, -1
[INFO ] ModelInfo - Loading model mlp on cpu()
[23:33:05] C:\source\mxnet\src\nnvm\legacy_json_util.cc:208: Loading symbol saved by previous version v1.5.0. Attempting to upgrade...
[23:33:05] C:\source\mxnet\src\nnvm\legacy_json_util.cc:216: Symbol successfully upgraded!
[23:33:05] C:\source\mxnet\src\engine\engine.cc:54: MXNet start using engine: NaiveEngine
[INFO ] ServingTranslatorFactory - Found translator in model directory: MyTranslator
[INFO ] ServingTranslatorFactory - Using translator: MyTranslator

[frankfliu]

Feel free to re-open this issue if you still have question.