Re-design lambda's to make sense w.r.t OEQ

[namiyousef]

Currently, the way the weights are specified is arbitrary. We can choose any lambda values of our choice, without any constraints on them. For example, one configuration has lambdas = [1, 1/100, 1/10000]

This can make sense, e.g. if the loss functions we choose are of O(1), O(100) and O(10000) respectively. However, in my opinion this choice is extremely arbitrary (note it isn't invalid per se). We need to choose losses such that they are all of the same order, then we can apply lambda's such that sum(lambdas) = 1 (thereby constraining the problem). Without this, I'm not sure if the OEQ makes sense?

After a quick search during our meeting today I found these:

• https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313
• https://blog.taboola.com/deep-multi-task-learning-3-lessons-learned/

Both seem to go for arbitrary losses. I would prefer going for a solution where the losses are of the same scale, allowing us to apply the constraint above. It would make more sense from an OEQ pov.

What are your thoughts?

[namiyousef]

Closing issue. Seems like scaling is arbitrary. The final code will have separate inputs for scaling and weight constraints separately.