Implement token masking to exclude BOS token

ApolloResearch / rib

Library for methods related to the Local Interaction Basis (LIB)

MIT License

2 stars 0 forks source link

Implement token masking to exclude BOS token #313

Open LuciusApollo opened 8 months ago

LuciusApollo commented 8 months ago

We would like to be able to make RIB graphs that exclude the first (BOS) token of the prompt from all basis and edge calculations. Could maybe implement optional masking of the activations over tokens to accomplish this.

stefan-apollo commented 8 months ago

Context: