Closed RissyRan closed 3 weeks ago
Clean up MoE brute force implementation, so that we have 2 strategies: 1) MoE dropless 2) MoE dropping (with expert parallelism in the future)
Test script 1: link Test script 2: link
Description:
Clean up MoE brute force implementation, so that we have 2 strategies: 1) MoE dropless 2) MoE dropping (with expert parallelism in the future)
Test
Test script 1: link Test script 2: link