Error issue using BatchPredict API

[suntingfeng]

Hi, I use BatchPredict API for batch prediction, and when the input data shape is NDList size: 1, 0 : (2, 64) int32 or NDList size: 2 0 : (64) int32，1 : (64) int32, it will appear RuntimeError: The size of tensor a (64) must match the size of tensor b (43) at non-singleton dimension 2, but shape is NDList size: 1 0 : (64) int32, it's ok. So I would like to ask if batchPredict can perform batch prediction？If possible, what format should be entered for the shape?

Thank you very much!

Criteria<NDList, Float> criteria =
                Criteria.builder()
                        .setTypes(NDList.class, Float.class)
                        .optModelPath(Paths.get(modelPath))
                        .optTranslator(new MyTranslator())
                        .optEngine("PyTorch")
                        .build();
ZooModel<NDList, Float> model = ModelZoo.loadModel(criteria);
predictor = model.newPredictor();

public static class MyTranslator implements Translator<NDList, Float> {
        @Override
        public Batchifier getBatchifier() {
            return Batchifier.STACK;
        }
        @Override
        public NDList processInput(TranslatorContext translatorContext, NDList inputs) { return inputs; }
        @Override
        public Float processOutput(TranslatorContext ctx, NDList outputs) {
            return outputs.get(0).getFloat();
         }
    }

predictor.batchPredict(Collections.singletonList(inputs));

[frankfliu]

@suntingfeng

batchPredict() function behavior depends on your Translator:

1. if Translator.getBatchifer() returns null (indicate it doesn't support batch), batchPredict() will invoke model forward one by one. In this case, you must manually add batch dimension to your input NDArray (e.g. for image shape(1,3,224,224))
2. If Translator.getBatchifer() return STACK, DJL will automatically stack multiple input (shape(3,224,224)) into (N,3,224,224), and then invoke forward as batch. the stack operator requires all NDArray has the same shape and type. This is what processInput() should do