-
Hi, I have read this good work and found it achieves wonderful performance. However, I have a question about the training process: Is the model trained from scratch on RSI data without any pretraining…
-
Hi, I pretrained the model on UCF101, and the linear evaluation on UCF101 is 74.0946%.
![1](https://user-images.githubusercontent.com/71969945/173597775-2230691d-dc28-45e8-8b23-832779bafba9.png)
…
-
I followed your BERT pretraining. However, after one week of training, the loss is still around 7.3. I use 8 GPU with 14 per batch. The rest is same as default.
INFO - 08/01/19 14:20:20 - 2:02:04 -…
-
I'd just like you to know that code with permissive licensing with attribution requirements **are possibly unsuitable for training set inclusion.** I'm bringing this to your attention not as a lawyer,…
ell1e updated
3 months ago
-
Could you tell me the complete training process of side tuning?
According to my understanding, your training process should be divided into the following stages:
- the pretraining of the base model
…
-
### Please check that this issue hasn't been reported before.
- [X] I searched previous [Bug Reports](https://github.com/OpenAccess-AI-Collective/axolotl/labels/bug) didn't find any similar reports.
…
-
Hi M. H. Kwon,
Your tokenization script is really helpful.
I trained a bert model with custom corpus using Google's Scripts like create_pretraining_data.py, run_pretraining.py ,extract_features.py…
-
Hi, thanks for your great work.
While running run_pretraining.py, I kept getting OOM for any size of the matrix.
I already reduce the batch size to 1 but didn't help.
I'm using 960M, TensorFlow-gpu…
-
In `deberta.mlm`, `MaskedLayerNorm ` is not imported from `deberta.ops`, and `PreLayerNorm` is undefined.
And I'm not sure if `deberta.mlm` contains codes for pretraining?
-
Thanks for your impressive work.
Can you share how to implement pretraining code?