if mlp_hidden_size is set it overrides mlp_ratio but the resulting config will still have mlp_ratio = 4 (the default value) which is confusing when looking at the config in e.g. wandb. I think we should either have mlp_ratio default to None or overwrite the config value via mlp_hidden_size if set
🐛 Describe the bug
if mlp_hidden_size is set it overrides mlp_ratio but the resulting config will still have mlp_ratio = 4 (the default value) which is confusing when looking at the config in e.g. wandb. I think we should either have mlp_ratio default to None or overwrite the config value via mlp_hidden_size if set
Versions
latest