where to find any experiments on real dataset？

kyegomez / LongNet

Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"

https://discord.gg/qUtxnK2NMf

Apache License 2.0

663 stars 63 forks source link

where to find any experiments on real dataset？ #15

Closed decoda-huanyi closed 8 months ago

decoda-huanyi commented 10 months ago

Where can I find some examples showing how to train longnet on a real dataset?

Upvote & Fund

We're using Polar.sh so you can upvote and help fund this issue.
We receive the funding once the issue is completed & confirmed by you.
Thank you in advance for helping prioritize & fund our backlog.

kyegomez commented 10 months ago

@decoda-huanyi Hey I'm still working on the actual model architecture, i need help with implementing it.

Would you like to help integrate it

xiehuanyi commented 9 months ago

I am acturally interested. Which dataset shall we start with? In my opinion, this dataset should have very long samples so that we can demostrate the power of longnet. I am not sure. Could we just try it on a noval or, you know, some other modals like music or images?

kyegomez commented 8 months ago

@xiehuanyi Hey we're about to start training on the enwiki dataset there's a competition for some money on whoever can train the best model on 1gb of wikipedia data!