KeyError on predict after re-loading a model from checkpoint

[jumutc]

After model is trained and periodically saved using ModelCheckpoint callback as in the following snippet:

# Define Callbacks
cb_lr = LearningRateScheduler(scheduler)
cb_mc = ModelCheckpoint(f'{model_timestamp}.model', monitor='val_tversky_crossentropy', save_best_only=True, verbose=3)
cb_es = EarlyStopping(monitor='tversky_crossentropy', mode='min', min_delta=0.0001, patience=10, verbose=3)
cb_tb = TensorBoard(log_dir="tensorboard", histogram_freq=0, write_graph=True, write_images=True)

model.evaluate(sample_list[20:], sample_list[3:20], epochs=epochs, callbacks=[cb_lr, cb_es, cb_tb, cb_mc])

model.load(f'{model_timestamp}.model', custom_objects={
              'dice_soft': dice_soft,
              'dice_crossentropy': dice_crossentropy,
              'tversky_crossentropy': tversky_crossentropy
          })
model.predict(sample_list[:4])

I get this error on predict on the latest master branch:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-14-a288f36ca86b> in <module>
      4 #               'tversky_crossentropy': tversky_crossentropy
      5 #           })
----> 6 model.predict(sample_list[:4])

c:\users\students\desktop\bacteria_cfu\code\miscnn-master\miscnn\neural_network\model.py in predict(self, sample_list, return_output, activation_output)
    172             pred_seg = np.concatenate(pred_list, axis=0)
    173             # Postprocess prediction
--> 174             pred_seg = self.preprocessor.postprocessing(sampleObj, pred_seg,
    175                                                         activation_output)
    176             # Backup predicted segmentation

c:\users\students\desktop\bacteria_cfu\code\miscnn-master\miscnn\processing\preprocessor.py in postprocessing(self, sample, prediction, activation_output)
    217         # Run Subfunction postprocessing on the prediction
    218         for sf in reversed(self.subfunctions):
--> 219             prediction = sf.postprocessing(sample, prediction)
    220         # Return postprocessed prediction
    221         return prediction

c:\users\students\desktop\bacteria_cfu\code\miscnn-master\miscnn\processing\subfunctions\resize.py in postprocessing(self, sample, prediction)
     71     def postprocessing(self, sample, prediction):
     72         # Access original shape of the last sample and reset it
---> 73         original_shape = sample.get_extended_data()["orig_resize_shape"]
     74         # Transform original shape to one-channel array
     75         prediction = np.reshape(prediction, prediction.shape + (1,))

KeyError: 'orig_resize_shape'

[muellerdo]

Related Commits: 629ced9b0e9cabc8e6e7e1f3e24a48943606d08e, 4cf3581cc20f678eaf48ce8eeb5e120058e6404a, 349bbc6f475ef362d0afef0d7e077af7d2878b9d, b49f07e87528737579fb566cf09073ae81be6747

Tasks

• [x] Reproduce bug
• [x] Fixed sample loading for postprocessing in predictions
• [x] Added new unittest for catching this scenario in the future
• [x] Tested locally and on TravisCI node
• [x] Merged dev branch into Master
• [x] Release new PyPI version

[muellerdo]

Hey @jumutc,

thank you for spotting this issue!

Apparently, a small bug of the sample loading in the postprocessing for predictions slipped through the latest bigger patch. I hoped that the hot fix solves this issue.

Could you test it out with the newest pypi/master version, if it works now?

Cheers, Dominik

[jumutc]

Hi @muellerdo, Sure, I will try! I have temporarily moved to 1.1.2 version but my Windows machine has issues installing from PyPi as there is always incompatibility of SimpleITK package for win distributions. I need to monkey patch this in setup.py. Can you maybe update the dependency for this package as well?

P.S. I'm a bit lazy and don't want to open a new issue for this

[jumutc]

@muellerdo I have tested master and all seems good although there are some unneeded print statements for predicted images shapes

[muellerdo]

• [x] Dependency Update of SimpleITK
• [x] Remove unnecessary print statement in Image IO Interface

Merged on current dev branch.

[muellerdo]

@muellerdo I have tested master and all seems good although there are some unneeded print statements for predicted images shapes

Great!