I wonder why the last_idx(the last index upto which blocks are chosen from for random attention) variable here has been set to 1024 even when the sequence length increases to 4096? Is this an error, or am I getting something wrong?
Thank you for your precious time.
Yours gratefully
I wonder why the
last_idx
(the last index upto which blocks are chosen from for random attention) variable here has been set to 1024 even when the sequence length increases to 4096? Is this an error, or am I getting something wrong?Thank you for your precious time. Yours gratefully