self.q_proj = MMoE(dim, self.att_experts, num_heads, dropout=0., concat=True)
self.out_proj = nn.ModuleList([
nn.Sequential(
nn.Linear(headdim, dim),
nn.Dropout(0.)
)
for in range(num_experts)
])
Where is this MMoE class defined? This class cannot be found in the project file
tankche1
commented
5 months ago
self.q_proj = MMoE(dim, self.att_experts, num_heads, dropout=0., concat=True) self.out_proj = nn.ModuleList([ nn.Sequential( nn.Linear(headdim, dim), nn.Dropout(0.) ) for in range(num_experts) ]) Where is this MMoE class defined? This class cannot be found in the project file