Closed vmarinowski closed 4 months ago
I'm working on adding that now 😄
Just added. Just be aware the masking isn't fully "causal" being the compressive memory updates allow tokens within a segment to attend to future tokens within that same segment. Without changing the compressive memory formulation, it's not something I can get around.
Just added. Just be aware the masking isn't fully "causal" being the compressive memory updates allow tokens within a segment to attend to future tokens within that same segment. Without changing the compressive memory formulation, it's not something I can get around.
If just use the fully causal self-attention without the compressive memory , would there be a significant decrease?
Good news. I was going back over the math and the code while working on another issue yesterday and realized that compressive memory doesn't break causality. It calculates its contribution to the final attention using information from the previous segment's tokens. So if you enable causal masking, the model is indeed fully causal. I meant to say something at the time, but got sidetracked.
Exactly where to use causal masking?