search

deephealthproject / eddl

European Distributed Deep Learning (EDDL) library. A general-purpose library initially developed to cover deep learning needs in healthcare use cases within the DeepHealth project.
https://deephealthproject.github.io/eddl/
MIT License
34 stars 10 forks source link

Upsampling layer #298

Closed lauracanalini closed 3 years ago

lauracanalini commented 3 years ago

Describe the bug Working with the use-case-pipeline, we notice an unexpected behavior, apparently due to the Upsampling layer. We reproduce the issue in EDDL modifying your test_onnx_upsample2D test.

If we train a network, evaluate it and save the onnx, and then import the onnx just saved and re-evaluate it, we obtain different predictions. Therefore, the calculated metric is slightly different between runs. 

RUN 1
sample 0 :0.000757227 0.000757227 7.60555e-07 7.60555e-07 4.54465e-09 4.54465e-09 4.19192e-12 4.19192e-12 6.32921e-14 6.32921e-14
sample 1 :0.00275645 0.00275645 3.17762e-05 3.17762e-05 2.05717e-06 2.05717e-06 2.49259e-08 2.49259e-08 1.41983e-09 1.41983e-09

RUN 2
sample 0 :0.000757227 0.000757227 0.000757227 4.15463e-05 4.15463e-05 7.60555e-07 7.60555e-07 4.54465e-09 4.54465e-09 4.19192e-12
sample 1 :0.00275645 0.00275645 0.00275645 0.000254507 0.000254507 3.17762e-05 3.17762e-05 2.05717e-06 2.05717e-06 2.49259e-08

If we replace the Upsampling layer with the Resize layer, the re-evaluation shows correct results instead. Of course we can't really use the Resize layer because we want to work with scale factor values in the network.

RUN 1
sample 0 :0.00109627 0.00109627 1.74091e-06 1.74091e-06 1.79723e-08 1.79723e-08 3.56336e-11 3.56336e-11 8.03426e-13 8.03426e-13
sample 1 :0.0034577 0.0034577 4.86836e-05 4.86836e-05 3.66603e-06 3.66603e-06 5.99838e-08 5.99838e-08 4.67307e-09 4.67307e-09

RUN 2
sample 0 :0.00109627 0.00109627 1.74091e-06 1.74091e-06 1.79723e-08 1.79723e-08 3.56336e-11 3.56336e-11 8.03426e-13 8.03426e-13
sample 1 :0.0034577 0.0034577 4.86836e-05 4.86836e-05 3.66603e-06 3.66603e-06 5.99838e-08 5.99838e-08 4.67307e-09 4.67307e-09

We tested with both master and development EDDL branches, both with CUDA and cuDNN.

To Reproduce Steps to reproduce the behavior:

  1. Replace the test_onnx_upsample2D.cpp with test_onnx_upsample2D.
  2. Run it - predictions of the two evaluating processes are different. (output_upsample)
  3. Replace the Upsampling layers with the Resize ones.
  4. Run it again - now predictions are the same. (output_resize)

Screenshots Qualitatively, in the pipeline, after the import from onnx file of SegNet with Upsampling layers, predictions seem to be translated.

chavicoski commented 3 years ago

Hi, This is now solved in develop branch. There was a problem with the UpSampling layer, because in ONNX the UpSample operator is deprecated, so we export the UpSample with a Resize operator in ONNX (which is the equivalent operator). And when loading the model, as the operator is a Resize, it is loaded as a Resize layer and the operation is not exactly the same than the UpSampling (that explains the differences in the predictions).

After the fix: The UpSampling layer is now a wrapper of the Resize layer, which is the one that performs the correct operation. Now there are no differences after importing the model.

Thanks for spotting the issue!