Open Hrancheng opened 2 months ago
Looks good!
There are some merge conflicts from recent updates.
I'll create a pull request from my gkielian repo to Hrancheng:CLA_test for weaving the latest changes in; PR will probably be arriving Thursday afternoon ish.
Update on time estimate, will be starting a PR for the edits tomorrow, and seems I'll probably have the PR for review on Monday
Looks great!
My review has some additional configuration settings and fields to add, afterwards should be ready to merge.
On a side note, if you can create slides about the approach and sections of the code modified to the team -- specifically the for loop with the block and high level of what was modified ( gpt_conf configuration param, train.py argparse, model.py attention and block) -- could be a good chance to show an in depth modification process.
Yes! I'll create slides along with codes modified next time to make this clear
@Hrancheng reviving interest in this due to potential hardware intersection, let's discuss early next week : )
Not sure whether this is correctly implemented, but a group of 4 attention layers(blocks) will use exactly the same K and V matrixes. We will work later to set this "4" to a user input argument.