search

deephealthproject / eddl

European Distributed Deep Learning (EDDL) library. A general-purpose library initially developed to cover deep learning needs in healthcare use cases within the DeepHealth project.
https://deephealthproject.github.io/eddl/
MIT License
34 stars 10 forks source link

Sensibly different loss / metric values in recent develop versions #242

Closed simleo closed 3 years ago

simleo commented 3 years ago

CC: @giobus75

Reported loss and metric values for examples in recent develop versions are sensibly different from the previous ones, and seem to change with the usage of low_mem, mid_mem, full_mem. For instance, at 494630f0, the output for mnist_mlp_train_batch was:

Loss: 0.00196105    Metric: 0.925063   |   
Evaluate:
Loss: 0.00091953    Metric: 0.96274   |   
Loss: 0.000922028   Metric: 0.9628   |

While now (74ffe859):

Loss: 0.248964  Metric: 0.925531   |   
Evaluate:
Loss: 0.114736  Metric: 0.963842   |   
Loss: 0.114752  Metric: 0.9638   |

And with low_mem or mid_mem setting for the computing service:

Loss: -nan  Metric: 0.0988916   |   
Evaluate:
Loss: -nan  Metric: 0.0980569   |   
Loss: -nan  Metric: 0.098   |
RParedesPalacios commented 3 years ago

Solved, I removed "that important line" by mistake....