`state` and `share_qk` options

[philgzl]

Hi again,

• Is the state argument of the different foward methods always None in your experiments? If not, when should it be set to something different from None?
• When using Retention, I can see you are sharing the query and key projection layers when RoPE is disabled here. Can you explain why? This does not seem to be explained in the paper.

[quancs]

• Is the state argument of the different foward methods always None in your experiments? If not, when should it be set to something different from None?

It's always None in our experiments.

When using Retention, I can see you are sharing the query and key projection layers when RoPE is disabled here. Can you explain why? This does not seem to be explained in the paper.

The sharing of query and key projection layers didn't degrade the performance in our experiments, but reduces the parameters and computational cost.

[philgzl]

Thanks!