VikParuchuri
commented
6 months ago For arxiv, I'd look at nougat or texify. Thanks for the paper pointer :)
Closed mrticker closed 6 months ago
VikParuchuri
commented
6 months ago For arxiv, I'd look at nougat or texify. Thanks for the paper pointer :)
For arxiv papers, specifically.
By the way, Vik, if you'd like a paper with challenging tables for testing marker, here's one: https://arxiv.org/abs/2311.15131
Here's what I get from a conversion:
Localizing Lying In Llama: Understanding Instructed Dishonesty On True-False Questions Through Prompting, Probing, And Patching
James Campbell? Cornell University jgc239@cornell.edu Richard Ren? University of Pennsylvania renrich@seas.upenn.edu
Abstract
Large language models (LLMs) demonstrate significant knowledge through their outputs, though it is often unclear whether false outputs are due to a lack of knowledge or dishonesty. In this paper, we investigate instructed dishonesty, wherein we explicitly prompt LLaMA-2-70b-chat to lie. We perform prompt engineering to find which prompts best induce lying behavior, and then use mechanistic interpretability approaches to localize where in the network this behavior occurs. Using linear probing and activation patching, we localize five layers that appear especially important for lying. We then find just 46 attention heads within these layers that enable us to causally intervene such that the lying model instead answers honestly.
We show that these interventions work robustly across many prompts and dataset splits. Overall, our work contributes a greater understanding of dishonesty in LLMs so that we may hope to prevent it.
1 Introduction
As large language models (LLMs) have shown increasing capability [Bubeck et al., 2023] and begun to see widespread societal adoption, it has become more important to understand and encourage honest behavior from them. Park et al. [2023] and Hendrycks et al. [2023] argue that the potential for models to be deceptive (which they define as "the systematic inducement of false beliefs in the pursuit of some outcome other than the truth"; Park et al. [2023]) carries novel risks, including scalable misinformation, manipulation, fraud, election tampering, or the speculative risk of loss of control. In such cases, the literature suggests that models may have the relevant knowledge encoded in their activations, but nevertheless fail to produce the correct output because of misalignment [Burns et al., 2022]. To clarify this distinction, Zou et al. [2023] delineates the difference between truthfulness and honesty: a truthful model avoids asserting false statements while an honest model avoids asserting statements it does not "believe." A model may therefore produce false statements not because of a lack of capability, but due to misalignment in the form of dishonesty [Lin et al., 2022]. Several works have since attempted to tackle LLM honesty by probing the internal state of a model to extract honest representations [Burns et al., 2022, Azaria and Mitchell, 2023, Li et al., 2023, Levinstein and Herrmann, 2023]. Recent black box methods have also been proposed for prompting and detecting large language model lies [Pacchiardi et al., 2023]. Notably, Zou et al. [2023] shows that prompting models to actively think about a concept can improve extraction of internal model representations. Moreover, in a context-following environment, Halawi et al. [2023] finds that there exists some "critical" intermediate layer in models, after which representations on true/false answers in context-following seem to diverge–a phenomenon they refer to as "overthinking." Inspired Phillip Guo? University of Maryland phguo@umd.edu by Halawi et al. [2023], we expand the scope from mis-labeled in-context learning to instructed dishonesty, wherein we explicitly instruct the model to lie. In this setting, we aim to isolate and understand which layers and attention heads in the model are responsible for dishonesty using probing and mechanistic interpretability approaches.
Our contributions are as follows:
2 Experimental Setup
Because we want to test dishonesty (or how the model 'intends' to answer, as opposed to whether it knows the answer in the first place), we compile an easy true/false dataset by taking the Azaria and Mitchell [2023] dataset and filtering for statements a smaller model would be most confident about. We do this by running LLaMA-2-7b-chat on a given datapoint (when prompting it to answer truthfully) and discarding it if it doesn't answer correctly (namely, the "True" or "False" token) with a probability > .85. We do this across all six splits of the dataset: "Cities", "Inventions", "Chemical Elements", "Animals", "Companies", and "Scientific Facts." Having compiled this dataset, we then use it to evaluate LLaMA-2-70b-chat along with various system prompts that either encourage it to tell the truth or lie. We input the true/false statements in a dialog context, wherein a user asks the model whether the statement is true or false. To determine the model's answer with a single token, we append a prefix to the model's answer so that the next most likely token is either "True" or "False".
We consider a model to act honestly if its accuracy on these true/false questions is significantly above random chance and dishonestly if its accuracy is significantly below random chance. We consider such behavior dishonest because in order answer with dramatically below-chance accuracy, the model must first accurately estimate the truth-value of the statement, but then actively negate it in its response. We find that inducing lying behavior can be surprisingly difficult and requires careful prompt engineering.
2.1 Probing
We train and test linear probes on the parallelized attention head activations (which we refer to as z activations, of dimension 128). These are binary logistic classifiers trained on the model's cached activations over the dataset, with labels corresponding to the truth value of the prompt. More details and definitions can be found in Appendix A.1.
2.2 Activation Patching
We also conduct activation patching, a technique from Meng et al. [2023] and Wang et al. [2022], on z activations. In activation patching, an intermediate activation is "patched" to have a different value (given by a different input) than it otherwise would have, affecting all future layers and the output of the model. This enables us to see the counterfactual effect of any given architectural component. In our setting, we store activations from the honest model's forward pass and edit them into the lying model's forward pass.
3 Results 3.1 Prompting For Dishonest Behavior
We find engineer prompts that successfully cause LLaMA-2-70b-chat to tell the truth and lie. Our prompts are illustrated in Table 1. Full prompts consist of 1) a system prompt, 2) a user prompt asking about the truthfulness of a statement, and 3) a prefix. We also report testing accuracy (recall) on both true and false statements, as opposed to a uniform accuracy metric, to ensure we can check if the model's output tends to be imbalanced. We measure performance by comparing the next token probability for "True" and "False". LLaMA-2-70b-chat is capable of lying, though it can be difficult to cause the model to output the incorrect answer on the token level. For example, the model appears to have a strong unbalanced bias toward outputting "False" instead of "True" on the token level on prompts inducing lying behavior, as seen in prompts 2-6 for "no prefix" in Figure 1 where accuracy on false statements is high and accuracy on true statements is low (in an ideal lying case, accuracy on all statements would be low). Empirically, we find that this is because when prompted to lie, the model tends to output filler tokens before "True" more often than for "False" (e.g. when model told to lie on a false statement, it generates "That's simple! Of course the statement is true..."; for a true statement, it generates "False! ...").
We overcome these issues by adding assistant response prefixes that reduce the observed biases.
For example, we may start the assistant's answer with "Oh that's an easy one! The statement is definitely...", which significantly improves the rate at which the model consistently outputs the incorrect answer when prompted to lie.
We demonstrate these prompt engineering subtleties in Figure 1. In each row, we evaluate a different combination of system prompt, user prompt, and prefix as defined in Table 1. Overall, our honest prompts can achieve accuracies upwards of 95% correct (high accuracy indicates excellent honest prompt performance), while our best liar prompts often get less than 5% correct (low accuracy indicates excellent lying prompt performance). Furthermore, the best lying prompts consistently and reliably negate the truth, as indicated by both positive and negative recall both being close to zero (few true positives or true negatives) - as opposed to failing to estimate truth properly, being confused and answering questions ambiguously, or being biased towards a certain token. By contrast, on all splits, random chance hovers around 50%. It's worth noting that it is much more difficult to instruct smaller models to output incorrect answers and act dishonestly. The prompts we use tend to perform poorly on LLaMA-2-7b-chat and LLaMA-2-13b-chat. For our experiments below, we indicate which prompts were used via the indexing scheme outlined in Table 1. For example, prompt 2aii refers to system prompt "Lying 2", user prompt "a", and prefix "ii".
System prompt 1 (Honest) is simply the standard LLaMA-2 system prompt as outlined in Touvron et al. [2023].
3.2 Honest-Liar Probe Transfer
We test the in-distribution and out-of-distribution transfer accuracy of all z activation probes at the last sequence position, across honest and liar system prompts. We also compare cosine similarities between probe coefficients as a proxy for similarities in representation. Figure 2a shows probes trained on one of the prompts (honest top row and liar bottom row) and tested on one of the prompts (honest left column and liar right column). The diagonal demonstrates the in-distribution accuracy of the probes, and the off-diagonal demonstrates transfer accuracy. Figure 2b similarly shows the cosine similarities between probe coefficients between the honest and lying prompts.
We find that both transfer probe accuracies and cosine similarities between honest and liar system prompts diverge at some intermediate layer; in the early-middle layers, a not-insignificant number of probes transfer with very high accuracy (reaching 90% chance) and discovered probe coefficients have very high cosine similarity. However, after an intermediate layer (around layer 23), many of the probes seem to reach very low (down to 10%) accuracy when transferred and the honest vs. liar probe coefficients become anti-parallel.
The Honest-Liar transfer suggests there are parallel representations before layer 23 and anti-parallel representations thereafter. This could be explained by the dishonest model first estimating the truth-value of the statement and then negating this representation around layers 19-23.
This "flip" in representation may be a result of the model's manipulation of the truth value at the last sequence position. This result mirrors the "overthinking" phenomenon found in Halawi et al. [2023].
Although they consider models who repeat mis-labeled data in a few-shot learning setting, one could see analogies between a model that has been implicitly instructed to repeat incorrect labels and a model that has been explicitly instructed to lie.
3.3 Activation Patching 3.3.1 Layer-Wise Patching
To further investigate this phenomenon, we patch in z activations from the honest model to the lying model to see if we can flip its predictions so that it instead answers honestly. We first test patching in groups of 5 contiguous attention layers on the last 25 sequence positions, finding that patching layers 19 through 23 is sufficient to get the lying model (whose base accuracy is 2%) to give the correct answer at a rate of 63% (shown in Figure 3). We also do patching for one to four contiguous layers, which can be found in Appendix B.3, where we again see the importance of layers 19-23.
3.3.2 Head-Level Patching
We next look to localize our activation patching further by finding a small set of attention heads that suffice to get the lying model to answer truthfully. After finding that layers 19-23 are the most important for lying, we decide to patch those five layers (from the honest to lying model) and find the most important heads within them. We do this by iterating through all attention heads in layers 19-23 and measure which heads' absence, i.e. lack of patching, causes the biggest drop in the patched model's accuracy. We patch the last 25 sequence positions, though the experimental setup can be replicated for any arbitrary number of sequence positions.
We run this exhaustive search over 50 samples from the Scientific Facts dataset split and on a baseline honest and liar prompt (starred in Table 2) and show our results in Figure 4. In particular, we see that the vast majority of heads in these layers have absolutely no effect on the model's accuracy on these 50 data points. On the other hand, there are 46 heads which do lower the patched model's accuracy when removed. Hence, we decide to perform activation patching with these 46 heads across 5 layers (which make up 0.9% of all heads in the network).
When we patch these 46 heads on the full Scientific Facts dataset split, we find that the patched model's accuracy improves (from 64% to 83%) over when we patched all heads across layers 19- 23. We suspect this might have to do with the presence of inhibitory heads (which when removed individually increase model accuracy as shown in Figure 4). Alternatively, it could be that too much blind patching results in a lower signal-to-noise ratio, motivating the need to test how well the selected heads generalize to other prompts and datasets.
3.3.3 Out-Of-Distribution Generalization
Given the same heads that were selected based on 50 samples from the Scientific Facts dataset split and on one prompt, we evaluate how robust their effect is across four varying prompts and on all six topic splits of the filtered Azaria and Mitchell [2023] dataset in Table 2. We test a limited set of prompts and patch the last 25 sequence positions; a table with results for many more prompts can be found in Appendix B.4, and comparisons of patching performance across more sequence positions can be found in Appendix B.5. The prompts we use to test generalization can be found in Table 1. Remarkably, we find strong generalization across both prompts and datasets (in spite of the few data points and single prompt used in selecting the 46 heads). In majority of prompt-dataset combinations, the "Patched Liar" - which was instructed to lie but patched with selected honest-prompted model head outputs - significantly recovers honest-prompted performance (>50% accuracy). We further find that one can patch both ways, as shown by the "Patched Honest" model - which was instructed to be honest but patched with selected liar-prompted model head outputs - nearing liar-prompted performance (<50% accuracy). In some cases, the patching also transfers remarkably well. For prompt 6fiii, the patching works very strongly, despite prompt 2fii being used to select the heads. In fact, on the Chemical Elements dataset split, the "Patched Liar" model actually gets higher accuracy than the honest model. On two other splits, the "Patched Liar" matches the honest model exactly, getting upwards of 98% accuracy. However, there exist some dataset/prompt combinations out of distribution that patching heads does not generalize as well on. On many of these combinations, both the honest and liar models tend to perform worse (without patching) at getting high and low accuracy respectively, which the model's performance might be inherently challenged in those particular contexts.
4 Conclusions And Future Work
We investigate a basic scenario of lying, in which we instruct an LLM to either be honest or lie about the truthfulness of a statement. Building on previous results that indicate activation probing can generalize out-of-distribution when prompted, our findings show that large models can exhibit dishonest behavior, in which they output correct answers if prompted to be honest and incorrect answers if prompted to lie. Nevertheless, we find this can require extensive prompt engineering given issues such as the model's propensity to output the "False" token earlier in the sequence than the "True" token. We obtain consistent prompted lying through prefix injection, and we then compare the activations of honest and dishonest models, localizing layers and attention heads implicated in lying.
We explore this lying behavior using linear probes and find that model representations between honest and liar prompts are quite similar in early-to-middle layers and then diverge sharply, becoming anti-parallel. This may provide evidence that a context-invariant representation of truth, as sought after by a collection of literature [Burns et al., 2022], ought to be found in earlier layers. Furthermore, we use activation patching to learn more about the mechanisms of individual layers and heads. Indeed, we find localized interventions that can fully correct the misalignment between the liar and honest-prompted models in either direction. Importantly, these interventions on just 46 attention heads show a reasonably strong level of robustness across datasets and prompts.
Dataset Split Prompt/Condition Facts Cities Companies Animals Inventions Elements Honest (prompt 1fii) 96.2%? 83.9% 98.4% 89.9% 81.1% 64.8% Patched Liar 83.0%? 68.8% 72.1% 63.5% 40.2% 46.0% Patched Honest 19.5% 48.2% 9.3% 13.1% 35.4% 40.0% Liar (prompt 2fii) 4.4%? 4.5% 3.1% 5.1% 19.7% 15.1% Honest (prompt 1fiii) 99.4% 99.1% 98.4% 97.8% 93.7% 89.2% Patched Liar 98.1% 99.1% 98.4% 94.2% 89.0% 91.4% Patched Honest 32.7% 41.1% 79.1% 59.1% 65.4% 64.0% Liar (prompt 6fiii) 2.5% 2.7% 0.8% 5.8% 7.9% 19.4% Honest (prompt 1fii) 96.2% 83.9% 98.4% 89.8% 81.1% 64.7% Patched Liar 78.0% 55.4% 60.5% 62.0% 39.4% 36.7% Patched Honest 18.2% 7.1% 3.9% 28.5% 35.4% 17.3% Liar (prompt 9fii) 2.5% 2.7% 1.6% 2.9% 12.6% 9.4% Honest (prompt 1fii) 96.2% 83.9% 98.4% 89.8% 81.1% 64.8% Patched Liar 88.7% 75.9% 97.7% 76.6% 61.4% 54.0% Patched Honest 71.1% 75.9% 95.3% 62.8% 85.0% 71.9% Liar (prompt 5fii) 8.2% 10.7% 2.3% 24.1% 14.2% 30.2%
While previous work has mostly focused on the truthfulness and accuracy of models that are honest by default, we zone in on lying by using an easy dataset and explicitly instructing the model to lie.
This setting has offered us valuable insights into the intricacies of prompting for dishonesty and the mechanisms by large models perform dishonest behavior. We hope that future work in this setting may give rise to further ways to prevent LLM lying to ensure the safe and honest use of LLMs in the real world.
Future Work
Our analysis is in a toy scenario—realistic lying scenarios will not simply involve the model outputting a one-token incorrect response, but could involve arbitrarily misaligned optimization targets such as swaying the reader's political beliefs [Park et al., 2023] or selling a product [Pacchiardi et al., 2023]. Future research may use methods similar to those presented here to find where biases/misalignments exist in the model and how more complex misalignments steer LLM outputs away from the truth. Furthermore, much more work should be done on analyzing the mechanisms by which the model elicits a truth-value representation and then on how the model uses this representation along with the system prompt to decide whether or not to respond truthfully. The observed representation flip could be a "truth" bit, "intent" bit, or could be related to more general behavior such as answer tokens or some inscrutable abstraction. Further mechanistic interpretability work testing the various truthfulness representations and heads discovered would enable stronger, more precise claims about how lying behavior works.
Acknowledgments And Disclosure Of Funding
We would like to thank EleutherAI for providing computing resources, Alex Mallen and Callum McDougall for their helpful advice, and the Alignment Research Engineer Accelerator (ARENA) program, where this project was born.
References
Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it's lying, 2023. Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form, 2023.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision, 2022. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing, 2023. Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. Overthinking the truth: Understanding how language models process false demonstrations, 2023. Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic AI risks, 2023. B. A. Levinstein and Daniel A. Herrmann. Still no lie detector for language models: Probing empirical and conceptual roadblocks, 2023. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model, 2023. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods, 2022. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT, 2023. Lorenzo Pacchiardi, Alex J. Chan, Sören Mindermann, Ilan Moscovitz, Alexa Y. Pan, Yarin Gal, Owain Evans, and Jan Brauner. How to catch an AI liar: Lie detection in black-box LLMs by asking unrelated questions, 2023. Peter S. Park, Simon Goldstein, Aidan O'Gara, Michael Chen, and Dan Hendrycks. AI deception: A survey of examples, risks, and potential solutions, 2023. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small, 2022. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to AI transparency, 2023.
Appendix A Further Experimental Setup A.1 Model Activations (Extended)
We utilize an autoregressive language model with a transformer architecture. We follow the multihead attention (MHA) representation set in Gurnee et al. [2023] and Li et al. [2023]. Given an input sequence of tokens X of length n, the model M : X ? Y outputs a probability distribution over the token vocabulary V to predict the next token in the sequence.
This prediction mechanism involves the transformation of each token into a high-dimensional space dmodel. In this paradigm, intermediate layers in M consist of multi-head attention (MHA) followed by a position-wise multi-layer perception (MLP) operation, which reads from the residual stream xi and then writes its output by adding it to the residual stream to form xi+1.
In MHA, the model computes multiple sets of Q, K, and V matrices to capture different relations within the input data. Each set yields its own self-attention output z. The specific attention head output z for any given head corresponds to the matrix of size dhead prior to undergoing a linear projection that yields the final self-attention output for the mentioned head. It can be conceptualized as a representation that captures specific relational nuances between input sequences, which might be different for each attention head. For this reason, while the MHA process is typically done with multiple sets of weight matrices with the results concatenated and linearly transformed, we train probes on the individual output activations z of each attention head, which has dimension dmodel/nheads = 8192/64 = 128.
It's important to note that while LLaMA-2-70b-chat utilizes a variant of the multi-head attention mechanism known as grouped-query attention (GQA), the fundamental principle remains similar to MHA. In GQA, key and value matrices are shared among groups of attention heads, as opposed to each head having its own distinct key and value matrices in standard MHA. This variation slightly alters the way attention is computed in intermediate steps, but does not significantly change the validity of methods that train probes on or activation patch the attention head output z.
B More Experiments B.1 Logit Attribution
We examine the logit attributions of the honest and the liar models, which is a technique for demonstrating how much each layer's attention directly contributes to the logit difference between the correct and incorrect logit ("True" - "False" or "False" - "True") by unembedding the attention output of every layer to the residual stream [Wang et al., 2022]. The main conclusion we can draw from this logit attribution is that layers before 40 do no or very little logit attribution, 40-45 start to do some, and 45-75 do the bulk of the logit attribution.
This seems to provide further evidence that the best truthful representations and lying preprocessing would not be found in the later layers, as the later layers merely seem to "write the model's output" and contain information about the model's response rather than the truth. Instead, the best truthful representations and the mechanisms for processing system prompts in order to lie are more likely to be found before most of the logit attribution is done, before layer 40.
B.2 Concept Erasure
Belrose et al. [2023] introduces a technique called concept scrubbing, specifically applied to intermediate activations in models. This method, known as LEACE (Least-Squares Concept Erasure), is designed to selectly remove specific types of information - in this case, linear truth information - from each layer of a model while perturbing the activations as little as possible. This permits us to analyze which truth representations the model actually makes use of. If concept scrubbing a particular set of layers causes the model to become much less accurate, it is likely that the model was relying on the linear truth information in those layers.
Given a concept defined by a classification dataset (X ? Rn×d, y ? {0, ..., k}n), LEACE can transform X such that no linear classifier can attain better than trivial accuracy at predicting the concept label from the transformed data (applying an affine transformation to each example depending on its class yi).
We specifically use Oracle LEAst-squares Concept Erasure, a variant of LEACE that uses test labels at inference time, to scrub as much linear truthful information as possible. However, due to varying lengths and information content in true/false statements, we choose to only apply O-LEACE to the last 15 sequence positions. Thus, not all linear truth information is erased across all sequence positions.
We run O-LEACE on both honest and lying models, attempting to erase the concept of truthfulness from each. As Figure 6 demonstrates, only a small number of layer range concept-erasures produce any noticeable change in model accuracy. Testing the erasure of five layers at a time, we find that task performance is most affected by O-LEACE on layers 19-23 (as well as 25-29), indicating a key role in processing truth-related information.
B.3 Layer-Wise Activation Patching
We show results for when we patch k layers on the Scientific Facts dataset split. For point i on the x-axis, we patch layers i through i + k. From left to right, we have k range from 1 to 5. In all cases, layer 19 seems especially prominent.
B.4 More Activation Patching Generalization Results For Patched Liar Model
The generalization of the activation patching technique was further assessed by testing its efficacy across many different prompts and datasets. This analysis was crucial to determine the robustness of the identified 46 attention heads in influencing the model's response towards honesty, irrespective of the initial prompt or the context of the dataset. Table 3 and Table 4 present extended results of this evaluation, showcasing how the patched model performed across different dataset splits and prompts.
The tables display the performance of the honest model, the liar model, and the patched liar model under various prompts. Each row corresponds to a specific combination of prompt and dataset.
The performance is measured in terms of accuracy - the percentage of responses that were correct for the given dataset and prompt. These results provide a comprehensive view of how well the patching technique generalizes across different contexts and how effective it is in aligning the model's responses with honesty.
These findings indicate that the patching of selected attention heads significantly improves the accuracy of the model's responses in most cases, even when tested on prompts and datasets that were not part of the initial selection process for these heads.
B.5 Activation Patching Across Sequence Positions
The effectiveness of activation patching was also evaluated across different sequence positions to understand its impact better. The performance was assessed by patching the heads at varying distances from the end of the sequence, specifically testing at the last 30, 25, 20, 15, 10, 5, and 1 sequence positions. The results, as summarized in Table 5, hint at how the model may process information across different stages of its sequence generation. For example, we found that patching the last 10 versus the last 5 sequence positions seems to make no difference in prediction accuracy, indicating that 46 heads identified may not conduct computation relevant to the prediction between sequence positions -10 and -5.
As expected, patching at earlier sequence positions (i.e. last 30, 25, 20 sequence positions) resulted in higher accuracy. Large drops in accuracy indicate regions that are likely involved in the initial stages of truth evaluation or processing the prompt's instruction for honesty/lying; this seems to occur between sequence position ranges [-15, -10] as well as [-5, -1].