aeon0
commented
6 years ago I can only speak for tensorflow backend: There you will not be able to stop the debugger in the loss function while training. That is because how tensorflow is built. Before training you are building a symbolic representation of the computational graph. For the actual training, tensorflow is using it's XLA compiler to compile the graph for faster computation (since python itself is quite slow). More info here: Tensorflow Graphs
You have different possibilities to debug the loss function: 1) Using tensorboard to visualize tensors 2) Using keras callbacks and K.eval(my_tensor) 3) Create dummy numpy array inputs and call compute_loss() with these numpy arrays. You can debug and use K.eval() to check the correct behavior.
What I have heard is, that with Theano it is easier to access the values while training. I think there is a print function which works during training. But have not tried that yet.
stavBodik
I am trying to debug compute_loss function , In my code the loss function :
def compute_loss(self, y_true, y_pred): """Compute mutlibox loss......Is passed to the compile function :
model.compile(optimizer=optimizer, loss=MultiboxLoss(NUM_CLASSES, alpha=MultiboxLoss_alpha, neg_pos_ratio=neg_pos_ratio, negatives_for_hard=MultiboxLoss_negatives_for_hard, class_weight=class_weight_train).compute_loss)Then as I understand right , this function is callable , and should be called inside the training function:
model.fit_generator(generator=gen_train.generate(True),....where can I find the lines which is executing the call to this function ? I cant stop there with debugger unfortunately because its callable function , like pointer to function in memory .
Please help ? Thanks !