LongNet can be used for fine-tuning large language models?

[mahuixian]

If I want to use LongNet to fine-tune the already trained large language model, can it be implemented？

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[kyegomez]

@mahuixian Add the masking strategy into the existing model's attention!