-
Hi, Deepspeed team! I am trying to run Bert-Pretraining with deepspeed. After preprocessing the wikipedia_en dataset and bookscorpus dataset, I specified the path in bert_large_lamb_nvidia_data.json, …
-
Running this on my Mac with CPU gives:
```
Dataset bookcorpus downloaded and prepared to /Users/arhamkhan/.cache/huggingface/datasets/bookcorpus/plain_text/1.0.0/eddee3cae1cc263a431aa98207d4d27fd8…
-
Consider shuffling bookcorpus:
```
dataset = nlp.load_dataset('bookcorpus', split='train')
dataset.shuffle()
```
According to tqdm, this will take around 2.5 hours on my machine to complete (ev…
-
### System Info
transformers==4.29.0
### Who can help?
_No response_
### Information
- [ ] The official example scripts
- [ ] My own modified scripts
### Tasks
- [ ] An officially supported tas…
-
Hi, gpauloski. Thanks for helping.
Recently, we have done some experiments, and the following questions need to be solved.
1. (**Without K-FAC, e.g., using LAMB in phase 1 and phase 2**): In p…
-
Hi, I am trying to use LLM-Pruner on Baichuan-13B model (https://github.com/baichuan-inc/Baichuan-13B). It is also llama structured so I thought it should work instantly, but I got some errors... I am…
-
Hmm, This may seem a bit excessive, but I'm a bit confused and don't know how to preprocess the data and train a RoBERTa model. Can you do a basic step by step tutorial for me?
Looks like I'm also l…
-
Wonderful work and thanks very much for your contribution!
I'm running the step 3.1 of corpus processing through the following command:
```
bash scripts/tools/process_full_doc_data_gpt2.sh ${BASE…
-
For a lot of configs in https://huggingface.co/datasets/sil-ai/bloom-speech, we get PreviousStepFormatError.
-