Could not find where the three "layers" are implemented in the code.

togethercomputer / MoA

Together Mixture-Of-Agents (MoA) – 65.1% on AlpacaEval with OSS models

Apache License 2.0

2.51k stars 346 forks source link

Could not find where the three "layers" are implemented in the code. #34

Closed mikegao2020 closed 1 month ago

mikegao2020 commented 1 month ago

Please clarify where does the aggregation happen in the "layer" level. Otherwise, this is quite similar to MOE. Thanks.

IsThatYou commented 1 month ago

Aggregation happens with the aggregator. This is different form MoE because we view each llm/agent abstractly as an "expert" rather than view FFN as experts