Open EwoutH opened 4 months ago
This issue was closed because it has been inactive for 14 days since being marked as stale.
Could this issue be reopened?
@ggerganov Have you look into this arch? A small Moe is interesting.
Can this request be reopen?
Very interesting new open source model just dropped, called JetMoE. It looks like the current SOTA as far as compute efficiency goes.
It has a very interesting model architecture: