Open cristianestojeda opened 1 year ago
Oh interesting. I'm curious why you'd find them useful. I was curious at how they'd look, but it was not a visual that is easily interpretable by humans. Ecco still collects the activations and you can plot them. That visual was using heatmaps to plot those values. That investigation turned to NMF factor analysis which brought forth more human-understandable patterns.
Could you shed some light on what you'd like to achieve? I might still have that code lying around somewhere. I couldn't find it from a quick search, but it's likely still around. I don't have a lot time to embed it into Ecco, but maybe just share it in its raw state if I find it.
Is it possible to include a method to visualize neuron activations and clustering neurons by activation values from https://jalammar.github.io/explaining-transformers/?
Some method to visualize this kind of figures: https://jalammar.github.io/images/explaining/activations-4.PNG https://jalammar.github.io/images/explaining/activations_1.PNG https://jalammar.github.io/images/explaining/activations_2.PNG
It would be very interesting to incorporate those visualizations as improve the interpretability of the Transformer architecture. Kind regards.