search

jalammar / ecco

Explain, analyze, and visualize NLP language models. Ecco creates interactive visualizations directly in Jupyter notebooks explaining the behavior of Transformer-based language models (like GPT2, BERT, RoBERTA, T5, and T0).
https://ecco.readthedocs.io
BSD 3-Clause "New" or "Revised" License
1.96k stars 167 forks source link

Request for feature addition: Visualization of Neuron Activations and Clustered Neuron Firings #93

Open cristianestojeda opened 1 year ago

cristianestojeda commented 1 year ago

Is it possible to include a method to visualize neuron activations and clustering neurons by activation values from https://jalammar.github.io/explaining-transformers/?

Some method to visualize this kind of figures: https://jalammar.github.io/images/explaining/activations-4.PNG https://jalammar.github.io/images/explaining/activations_1.PNG https://jalammar.github.io/images/explaining/activations_2.PNG

It would be very interesting to incorporate those visualizations as improve the interpretability of the Transformer architecture. Kind regards.

jalammar commented 1 year ago

Oh interesting. I'm curious why you'd find them useful. I was curious at how they'd look, but it was not a visual that is easily interpretable by humans. Ecco still collects the activations and you can plot them. That visual was using heatmaps to plot those values. That investigation turned to NMF factor analysis which brought forth more human-understandable patterns.

Could you shed some light on what you'd like to achieve? I might still have that code lying around somewhere. I couldn't find it from a quick search, but it's likely still around. I don't have a lot time to embed it into Ecco, but maybe just share it in its raw state if I find it.