Closed MicheleCancilla closed 3 years ago
Thank you! We'll check it soon
Hello, we recently fixed a bug in the batchnorm inference mode that may be causing this loss in the accuracy. The fix is not in the master branch yet, but if you can, use the develop branch and repeat the experiment to see if it still happens.
That's good to know! @chavicoski Do the ONNX models that come from PyTorch need to be "simplified" with tools like onnx-simplifier? I didn't do that and EDDL loaded them with no errors/warnings. I exported the ONNX this way:
dummy_input = torch.ones(4, 3, args.size, args.size, device='cpu')
model.train()
torch.onnx.export(model, dummy_input, f'{args.model}.onnx', verbose=True, export_params=True,
training=torch.onnx.TrainingMode.TRAINING,
opset_version=12,
do_constant_folding=False,
input_names=['input'],
output_names=['output'],
dynamic_axes={'input': {0: 'batch_size'}, # variable length axes
'output': {0: 'batch_size'}})
No, if the eddl loads the model you don't need to apply the onnx simplifier. In case that the load funcion of eddl fails, you can try to apply the onnx simplifier to avoid some operators that we don't support. But as I said, for this case you don't need it.
I repeated tests with EDDL develop (457a71d).
EDDL | PyTorch |
---|---|
![]() |
![]() |
Results are good as expected (I did not use cuDNN), there are perhaps more spikes compared to PyTorch. Thanks for the tips😉
We are encountering problems at improving our UC12 classification pipeline performance.
We changed the code to load a ResNet model pretrained on ImageNet (the ONNX file is exported by PyTorch), then substitute the last layer and fine-tune the whole network on our data. This procedure runs fine with PyTorch (80% validation accuracy: resnet18 after 60 epochs, resnet50 after 20 epochs), while EDDL hangs at ~0.60 accuracy. I made two charts showing the evolution of the accuracy over time. Both pipelines (PyTorch and EDDL) use same model (the same ONNX file), augmentations, loss, metric, and hyperparameters.
We thought that something was wrong on input samples augmented by ECVL, but they look good and augmented as expected (we used this EDDL-based code to inspect them), so we think that there is an error somewhere else.
Could you please have a look at the code skin_lesion_classification_training_reproducible.cpp to find out what is wrong?
How to reproduce:
EDDL pipeline
prerequisites:
PyTorch pipeline
Comparison
Test made with EDDL 0.9.1b and ECVL 0.3.4. eddl_pt_comparison.xlsx