Remove duplicate code between GPT2 and CoCa

spravil commented 7 months ago

luzian-hahn commented 7 months ago

Some points to that after @flxst and me have looked a little into GQA:

We found that both, CoCa but also GPT2 are (by today) implemented such that they are using one "central" config object, from which they both initialize there sub-components relevant for the respective model objects. Since this problem here also would involve some kind of refactoring, it might make sense to have a similar nested structure "re-using" analogue sub-components (like the attention implementation).
right now, we mimicked the test infrastructure of your changes, @spravil in the CoCa-branch but have the attention utility under "models". It would be good to unify this as well, but this is probably implicitly included within the work in the first point.
We are currently doing some config validations, but mostly in the different component's code, not in the related config objects. (mostly due to the fact, that there is no alignment between sub-components and sub-configs yet). We might wanna think about a unified way to do this validations. Those validations mostly take care of picking valid head-counts in relation to the embeddingsize. We also have to think about having those checks at the configs but also the sub-components, since modalities might also be used as library and people use the sub-components directly instead of approaching them via dedicated config objects.

flxst commented 7 months ago

We should probably work on this as soon as the related PRs (#16 and #74) are merged into main.

luzian-hahn commented 7 months ago

We should probably work on this as soon as the related PRs (#16 and #74) are merged into main.

Good point. We can maybe talk about it on monday during the hackathon

mali-git commented 4 months ago

@spravil, what is the status here?

Modalities / modalities