rhymes-ai / Aria

Codebase for Aria - an Open Multimodal Native MoE
Apache License 2.0
375 stars 20 forks source link

Question about clustering-based packing #21

Open Hannibal046 opened 2 days ago

Hannibal046 commented 2 days ago

Hi Teams, Congratulate on this impressive model! I have a question about the language model packing strategy discussed in the paper. Currently, I am also trying something similar to https://arxiv.org/abs/2310.10638, but no luck yet. Could you please share more details and insights about this method? Thanks! image