search

databricks / megablocks

Apache License 2.0
1.11k stars 154 forks source link

the wrong loss func was chosen at evaluation #93

Open peterjc123 opened 5 months ago

peterjc123 commented 5 months ago

The loss func is always moe_loss_func as can be seen here. But the loss is only calculated when training, which can be seen here. We should fallback to the original loss func during evaluation.

tgale96 commented 5 months ago

That sounds reasonable to me! We'd be happy to take a PR if you'd like to fix this in our Megatron-LM fork!

peterjc123 commented 5 months ago

@tgale96 I've put up a fix. Please take a look when you have time. https://github.com/stanford-futuredata/Megatron-LM/pull/6