[Question] loading Llama3-8B-instruct to HookedTransformer got a warning saying You are not using LayerNorm, so the writing weights can't be centered! Skipping!

[wangyifei0047]

Question

My gpu can't download checkpoints directly from huggingface.co, I write a function to load models from local directory, then passing to HookedTransformer. As a result, it jumps a warning, I am a little confused, I raise a few questions: -whether this function configurations are proper; -whether this warning influences the results of mechanistic interpretability

[ArthurConmy]

Read this: https://github.com/TransformerLensOrg/TransformerLens/blob/main/further_comments.md#centering-writing-weights-center_writing_weight

The warning will have no impact on your results. Feel free to pass center_writing_weights=False to the from_pretrained call to stop the warning yapping at you