nodefluxio / vortex

A Deep Learning Model Development Framework for Computer Vision
27 stars 6 forks source link

[BUG] Runtime validation dataloading bug #53

Closed triwahyuu closed 4 years ago

triwahyuu commented 4 years ago

Describe the bug The last data on onnx runtime validation causing an error when using batch size > 1

/home/triwahyuu/work/vortex-gh/src/runtime/vortex/runtime/onnx/onnxruntime.py:49: UserWarning: disabling onnx runtime fallback
  warnings.warn("disabling onnx runtime fallback")
Traceback (most recent call last):                                                                                                                                                                                 
  File "src/development/vortex/development/vortex_cli.py", line 40, in <module>
    main()
  File "src/development/vortex/development/vortex_cli.py", line 37, in main
    eval(selected_stage).main(args)
  File "/home/triwahyuu/work/vortex-gh/src/development/vortex/development/ir_runtime_validate.py", line 32, in main
    eval_results = validation_executor.run(batch_size=args.batch_size)
  File "/home/triwahyuu/work/vortex-gh/src/development/vortex/development/core/pipelines/validation_pipeline.py", line 173, in run
    eval_result = validator()
  File "/home/triwahyuu/work/vortex-gh/src/development/vortex/development/core/engine/validator/base_validator.py", line 357, in __call__
    results = self.predict(image=image)
  File "/home/triwahyuu/work/vortex-gh/src/development/vortex/development/core/engine/validator/base_validator.py", line 211, in predict
    image=image, *args, **kwargs,
  File "/home/triwahyuu/work/vortex-gh/src/development/vortex/development/core/engine/validator/base_validator.py", line 327, in runtime_predict
    result = predictor(image,*args, **kwargs)
  File "/home/triwahyuu/work/vortex-gh/src/runtime/vortex/runtime/basic_runtime.py", line 96, in __call__
    outputs = self.predict(*args, **predict_args)
  File "/home/triwahyuu/work/vortex-gh/src/runtime/vortex/runtime/onnx/onnxruntime.py", line 81, in predict
    self.output_name, run_args
  File "/home/triwahyuu/virtualenvs/dev/lib/python3.6/site-packages/onnxruntime/capi/session.py", line 142, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: input for the following indices
 index: 0 Got: 2 Expected: 16
 Please fix either the inputs or the model.

To Reproduce Steps to reproduce the behavior:

  1. export onnx model with batch size > 1
  2. run a runtime validation on it

Additional context this is caused by the last batch from the data is not always have the same size as the batchsize