-
When resuming finetuning, I see that the CycleIterator is forwarded to the dataset where the iteration is to continue from:
https://github.com/Lightning-AI/litgpt/blob/f3343784bbd192490e2a70aa5ef75…
-
-
Hi M. H. Kwon,
Your tokenization script is really helpful.
I trained a bert model with custom corpus using Google's Scripts like create_pretraining_data.py, run_pretraining.py ,extract_features.py…
-
Related issues:
- #24
- #40
-
Hi, I pretrained the model on UCF101, and the linear evaluation on UCF101 is 74.0946%.
![1](https://user-images.githubusercontent.com/71969945/173597775-2230691d-dc28-45e8-8b23-832779bafba9.png)
…
-
Hi,
When running wiki103 gpt2-m and gpt2-l baseline pretraining experiments,
`python run.py experiment=wt103/gpt2m`
and
`python run.py experiment=wt103/gpt2l`
will receive non-converge error.
…
-
Thanks for your impressive work.
Can you share how to implement pretraining code?
-
Hi, I have read this good work and found it achieves wonderful performance. However, I have a question about the training process: Is the model trained from scratch on RSI data without any pretraining…
-
Dear all,
please excuse the potentially silly beginner question.
I want to train a model and am using `litgpt pretrain ...`
In the corresponding `pretrain.py` file, it's hardcoded that FSDP i…
-
As the title implies, can you release both pretraining and fine-tuning log?