Questions about harcoded value in L2 regularization loss added to the total loss

[philferriere]

In the "base" OSVOS implementation shared here, the total loss is computed as follows:

    # Define loss
    with tf.name_scope('losses'):
        if supervison == 1 or supervison == 2:
            dsn_2_loss = class_balanced_cross_entropy_loss(end_points['osvos/score-dsn_2-cr'], input_label)
            tf.summary.scalar('dsn_2_loss', dsn_2_loss)
            dsn_3_loss = class_balanced_cross_entropy_loss(end_points['osvos/score-dsn_3-cr'], input_label)
            tf.summary.scalar('dsn_3_loss', dsn_3_loss)
            dsn_4_loss = class_balanced_cross_entropy_loss(end_points['osvos/score-dsn_4-cr'], input_label)
            tf.summary.scalar('dsn_4_loss', dsn_4_loss)
            dsn_5_loss = class_balanced_cross_entropy_loss(end_points['osvos/score-dsn_5-cr'], input_label)
            tf.summary.scalar('dsn_5_loss', dsn_5_loss)

        main_loss = class_balanced_cross_entropy_loss(net, input_label)
        tf.summary.scalar('main_loss', main_loss)

        if supervison == 1:
            output_loss = dsn_2_loss + dsn_3_loss + dsn_4_loss + dsn_5_loss + main_loss
        elif supervison == 2:
            output_loss = 0.5 * dsn_2_loss + 0.5 * dsn_3_loss + 0.5 * dsn_4_loss + 0.5 * dsn_5_loss + main_loss
        elif supervison == 3:
            output_loss = main_loss
        else:
            sys.exit('Incorrect supervision id, select 1 for supervision of the side outputs, 2 for weak supervision '
                     'of the side outputs and 3 for no supervision of the side outputs')
        total_loss = output_loss + tf.add_n(tf.losses.get_regularization_losses())
        tf.summary.scalar('total_loss', total_loss)

The lesion segmenter uses the same L2 regularization as in the reference implementation. However, the liver segmenter uses the following instead:

        # total_loss = output_loss + tf.add_n(slim.losses.get_regularization_losses())
        total_loss = output_loss + tf.add_n(tf.losses.get_regularization_losses())
        tf.summary.scalar('losses/total_loss', total_loss)

        # total_loss = output_loss + 0.001 * tf.add_n(slim.losses.get_regularization_losses())
        total_loss = output_loss + 0.001 * tf.add_n(tf.losses.get_regularization_losses())

        tf.summary.scalar('losses/total_loss', total_loss)

Why is this 0.001 scaling necessary in the liver segmenter? What procedure did you use to come up with this scaling factor?

Thank you for your help clarifying this!

[miriambellver]

We used this scaling factor as the range of our loss differed from the range of the loss of OSVOS.

[philferriere]

To be perfectly clear, was this a value you chose arbitrarily, or did you use a specific procedure to compute this value?

Thank you again for your help with those implementation details!

[miriambellver]

We just observed the range of the losses and how different they were compared to OSVOS, and thought that this value was more convenient. We didn't do any hyper-parameter search.

[philferriere]

Got it. Thank you for your help!