-
Your docs state that you pretrained the ResNet backbones using the Masked Autoencoding (MAE) SSL method. I was under the assumption that MAE was exclusive to Vision Transformers (ViT) due to the patch…
-
I'm having trouble getting the raw dataset from CuBERT. Can you release the dataset for pretraining?
-
Hello, I found that during the pre-training process, the memory occupied keeps increasing in the iteration process, I want to know why this is, is the same for your training process and how much memor…
-
I got this error when doing further-pretraining
my environment
Ubuntu 18.04.4 LTS (GNU/Linux 5.4.0-74-generic x86_64)
GPU 2080ti
I use following command
python run_pretraining.py \
--inp…
-
## ❓ Questions and Help
According to the paper, it only does sentence permutation and span masking (ignoring the other noises that BART is pretrained with). I found [something](https://github.com…
-
Hi authors, thank you for this impressive work.
Is it possible to provide a pretraining script and a small sample of the processed data used for pretraining? I would like to try pretraining a model…
-
I find some problems to obtain the pretraining reddit datasets. Could u release the pretraining datasets to help me to re-pretrain DialogVED? Thanks a lot.
-
As far as I know, one can pretrain fine using TinyStories:
```python
litgpt/pretrain.py --data litgpt.data.TinyStories
```
Should we add this to the documentation?
Right now, we only have …
rasbt updated
3 months ago
-
Hi,
Do you have the entire pre training dataset or did you randomly sample ZINC15 and ChemBl ? If so what tranches?
-
I have a few different pretraining tasks that can be incorporated with MLM loss objective. We can run parallel jobs for pretraining using different loss functions. The objective is to gain understandi…