This PR adds an 4D parallelism example of using veScale to run a Mixtral 8x7B model that is directly imported from HuggingFace without any model code modifications.
At the same time, we also develop a debug utility of printing logs and reorganize some code structures in this PR.
This PR adds an 4D parallelism example of using veScale to run a Mixtral 8x7B model that is directly imported from HuggingFace without any model code modifications.
At the same time, we also develop a debug utility of printing logs and reorganize some code structures in this PR.