search

openai / transformer-debugger

MIT License
4.01k stars 231 forks source link

Confusion about the MLP neuron index range #27

Closed ptyin closed 3 months ago

ptyin commented 3 months ago

As said in datasets.md,

https://github.com/openai/transformer-debugger/blob/dc1f898725113bec6cf1006e48f9c5219f8fbdde/datasets.md?plain=1#L17

But I believe the neuron size is should be 4 times d_model which is 3072 instead of 3084. Could you kindly explain the number here?

WuTheFWasThat commented 3 months ago

that's an error in the documentation, feel free to make a PR to fix :)