-
-
As far as I know, one can pretrain fine using TinyStories:
```python
litgpt/pretrain.py --data litgpt.data.TinyStories
```
Should we add this to the documentation?
Right now, we only have …
rasbt updated
3 months ago
-
In Section 4.1, it seems that only IN1K is used for pretraining. But in Table 1, both SA-1b and IN1K are used in pretraining. Which is the correct one?
Cbtor updated
6 months ago
-
Really nice work! I have questions about the pretraining of ViT. The paper mentions that the ViT is pretrained on publicly available histology datasets through SSL. I am wondering where I can find the…
-
Unexpected key(s) in state_dict: "layers.0.estimation_gate.FC1.weight", "layers.0.estimation_gate.FC1.bias", "layers.0.estimation_gate.FC2.weight", "layers.0.estimation_gate.FC2.bias", "layers.0.inh_l…
-
I was wondering if there are sample packing approaches defined somewhere for preprocessing and tokenization of datasets? I looked through different prepare_*.py, but couldn't find anything related to …
-
Hello,
Thank you for releasing the codes for pretraining MPNet!
I am trying to continue training of the language model task on a custom dataset from the released checkpoint using the ``--restore-fil…
-
I have a few different pretraining tasks that can be incorporated with MLM loss objective. We can run parallel jobs for pretraining using different loss functions. The objective is to gain understandi…
-
Your work is great. Can you provide the pre-training model for testing? I want to test it. Thank you very much!
yy2yy updated
4 years ago
-
,,,,
taeil updated
3 years ago