search

NifTK / NiftyNet

[unmaintained] An open-source convolutional neural networks platform for research in medical image analysis and image-guided therapy
http://niftynet.io
Apache License 2.0
1.36k stars 404 forks source link

KeyError: 'total_loss' in deep_boosted_regression application #454

Closed hasansari closed 4 years ago

hasansari commented 4 years ago

I have been trying to run the Deep Boosted Regression with some 3D datasets and get the following error after the first validation step. The training runs fine until then.

INFO:niftynet: validation iter 200, Loss=710.3148193359375, data_loss1=238.03733825683594, data_loss2=236.62986755371094, data_loss3=235.64764404296875 (79.673389s) INFO:niftynet: cleaning up... INFO:niftynet: iter 200 saved: //autofs/space/atticus_001/users/hs584/Deeplearning/niftynet_env/NiftyNet/niftynet/models/pelvis_mrct/models/model.ckpt INFO:niftynet: stopping sampling threads ERROR:niftynet: Session failed to close after 30 seconds. Continuing after this point may leave your program in an undefined state. Traceback (most recent call last): File "net_run.py", line 8, in sys.exit(main()) File "/autofs/space/atticus_001/users/hs584/Deeplearning/niftynet_env/NiftyNet/niftynet/init.py", line 147, in main app_driver.run(app_driver.app) File "/autofs/space/atticus_001/users/hs584/Deeplearning/niftynet_env/NiftyNet/niftynet/engine/application_driver.py", line 205, in run loop_status=loop_status) File "/autofs/space/atticus_001/users/hs584/Deeplearning/niftynet_env/NiftyNet/niftynet/engine/application_driver.py", line 331, in loop ApplicationDriver.loop_step(application, iter_msg) File "/autofs/space/atticus_001/users/hs584/Deeplearning/niftynet_env/NiftyNet/niftynet/engine/application_driver.py", line 366, in loop_step ITER_FINISHED.send(application, iter_msg=iteration_message) File "/homes/8/hs584/.local/lib/python3.6/site-packages/blinker/base.py", line 267, in send for receiver in self.receivers_for(sender)] File "/homes/8/hs584/.local/lib/python3.6/site-packages/blinker/base.py", line 267, in for receiver in self.receivers_for(sender)] File "/autofs/space/atticus_001/users/hs584/Deeplearning/niftynet_env/NiftyNet/niftynet/engine/handler_performance.py", line 32, in update_performance_history current_loss = console_content['total_loss'] KeyError: 'total_loss'

Any possible solutions for this? Thanks a lot

danieltudosiu commented 4 years ago

I am not the maintainer of that application, but looking into it it just misses the total_loss from the TF_SUMMARIES.

Please try and replace 

            outputs_collector.add_to_collection(
                var=loss, name='LossSum',
                average_over_devices=True, summary_type='scalar',
                collection=TF_SUMMARIES)

with:

            outputs_collector.add_to_collection(
                var=loss, name='total_loss',
                average_over_devices=True, summary_type='scalar',
                collection=TF_SUMMARIES)