debug test.py returns zero output - determine why deconvnet and unet model snapshots produce zero output (same class predictions)

microsoft / seismic-deeplearning

Deep Learning for Seismic Imaging and Interpretation

MIT License

402 stars 142 forks source link

debug test.py returns zero output - determine why deconvnet and unet model snapshots produce zero output (same class predictions) #304

Closed yalaudah closed 4 years ago

yalaudah commented 4 years ago

Possible culprits:

Saving models in Ignite? Serialization issues?
Possible bug in test.py?

Update: the model is definitely trained correctly, and the issues does not seem to be related to the serialization. Its definitely in the test.py code. I've ran it with an Imagenet pretrained model and it still returned zeros.

Rescoping to just Unet - make sure this works. Table HRNet for later.

The ask is to either repro with UNet and fix or label as not reproducible and close the work item.

yalaudah commented 4 years ago

Update: this is an HRNet related issue for sure. I couldn't duplicate the error with UNet.

Lets try this with:

[x] UNet
[x] SEResNet
[x] DeconvNet