search

rwth-i6 / returnn_common

Common building blocks for RETURNN configs, such as models, training concepts, etc
7 stars 4 forks source link

mark_as_loss: extension for LR scheduling? #211

Closed albertz closed 1 year ago

albertz commented 2 years ago

LR scheduling such as Newbob depends on some loss measure. In RETURNN, this can be configured via learning_rate_control_error_measure.

How do we handle this here? We could extend mark_as_loss with a flag like main_lr_key but not sure if that is a good idea. Esp, what happens if this is called multiple times? Error? Ignore and only take latest? Combine all?

We already enforce the user to provide an explicit name for each loss, so it would not necessarily be needed, as the user can just use the same name then for learning_rate_control_error_measure. But this is maybe prone to errors. Esp, when the user has an error there, this error would get unnoticed, as RETURNN has some fallback logic in case learning_rate_control_error_measure does not match. Maybe there can be an additional flag on RETURNN side, like learning_rate_control_error_measure_must_match = True, which disables the fallback logic, and throws an error when the key does not match?

albertz commented 1 year ago

We could extend mark_as_loss with a flag like main_lr_key

I don't like this idea too much. We added the mark_as_loss name option (#207) also for the LR scheduling, so we don't really need it.

I think actually nothing really needs to be done anymore. The user has full control over it.