search

johnmarktaylor91 / torchlens

Package for extracting and mapping the results of every single tensor operation in a PyTorch model in one line of code.
GNU General Public License v3.0
487 stars 17 forks source link

AttributeError: 'Parameter' object has no attribute 'tl_tensor_label_raw' #15

Closed miccio-dk closed 1 year ago

miccio-dk commented 1 year ago

Hi, and thank you for this amazing tool!!

I'm trying to visualize a small model that implements a custom quantization scheme. I'm particularly interested in visualizing both the input/outputs of the model, as well as I made a little notebook showcasing the error in its entirety: https://gist.github.com/miccio-dk/e3ee36cbe7c3ce8f2a48e1f07df65a95

I hope the notebook provides sufficient context/information, otherwise feel free to reach out for more info :)

johnmarktaylor91 commented 1 year ago

Hello, thanks so much for the kind words, and thanks for describing this so carefully! There was a small bug to fix, but now TorchLens is able to visualize your model. I'll push the updated version to pip as soon as the test suite finishes running.

image

That said, the bug is actually a potentially informative one. So far for TorchLens, parameters are treated differently from regular tensors, in that I anticipated them basically "staying put" rather than being passed in and out of modules and so on. For example, in the visualization rather than having their own separate nodes, the parameters for a given operation are just shown as text for the associated operation (e.g., for convolution the dimensions of the weights and biases are labeled on the node for convolution, rather than being their own separate boxes; this seemed less visually cluttered to me).

The fact that parameters are assumed to stay put was the source of the bug here, since your model has parameters being passed into the forward call of a module. So perhaps I can ask the question--for the use cases that you anticipate, would it be useful for parameters to be marked as their own nodes? This would require a bit of a refactor, so I wouldn't be able to roll this out right away, but I can bump it up the to-do list if it'd be useful for you.

EDIT: Also, quick clarification--by "layer variable names", do you mean the attribute names of your modules? For example, "qw", "qi", "qfc1", etc.

miccio-dk commented 1 year ago

Amazing, thank you so much again for the super fast fix :) The graph you're showing above is exactly what I needed for my specific use case, so I wouldn't worry about marking the parameters as their own nodes, unless it looks very weird at higher depths. Regarding the EDIT: that's exactly what I meant, yes. It's super useful to be able to see the names of these variables as they appear in the code.

johnmarktaylor91 commented 1 year ago

Awesome, glad it was a quick fix--and thanks again for pinning this down, it's definitely an edge case I wouldn't have thought of from the armchair ;) Just pushed the updated version to pip. Always open to further feedback on how I can make it more helpful for you.