-
I was wondering if there are sample packing approaches defined somewhere for preprocessing and tokenization of datasets? I looked through different prepare_*.py, but couldn't find anything related to …
-
Great job!
Hello , i wonder if you can tell me the training mlm accuracy of encoder and decoder. Im training my retromae model now.
wwx13 updated
2 months ago
-
Hi, are you going to release pretraining code?
-
Hey,
I was looking into the paper as I want to replicate the work. In the data preparation step it is mentioned **PMC** and **IDL** data is used for pre-training the model, are both data sources us…
-
A colleague of mine made the comment I should pretraing the models to yield more robust models and
better accuracy. Now... how can I do that - or ... what are possible avenues here?
My models are …
-
run_distill.py
```python
from datasets import build_pretraining_dataset, build_multi_pretraining_dataset
from engines.engine_for_distill import train_one_epoch
from utils import NativeScalerWit…
-
Since my application involves non-natural images, I aim to pre-train the model on a domain-specific dataset. However, I haven't found any relevant tutorials for this task. I would greatly appreciate i…
-
## 🐛 Bug
Loss goes to very low values and accuracy is 1 after several updates.
I'm sure it's bug and this metrics are wrong.
### To Reproduce
1. Get any considerable amount of wavs (2k hours i…
-
Thank you for open-sourcing the code! I didn't find descriptions about pretraining datasets in the paper. Was Starmie pertained on benchmark datasets?
-
Hello,
Thanks a lot for you great work ! We were really impressed !
We wanted to know where we could download Naccos ? It seems that the link is broken: https://github.com/qanastek/DrBERT/blob/main
…