-
Thank you for the detailed report for Llama3.1, which is very inspirational. I read the report and have a doubt about training infrastructure. In chapter 3.3.2 titled _Parallelism for Model Scaling_.…
-
### ❓ The question
do you know the slurm script for configs/official/OLMo-7B.yaml?
looking for multi-node slurm script
-
# 🐞 Bug
The same batch size, different micro batches, the algorithm effects are inconsistent.
I have fixed the random seed.
I set chunks equal to 2 or 4
## Code that reproduces
```python
imp…
-
### What is the problem the feature request solves?
Query:
```sql
select ss_sold_date_sk, ss_sold_time_sk, ss_quantity, d_year, d_moy, d_dom
from date_dim join store_sales on d_date_sk = ss_so…
-
Hi,
I've been experimenting with GPipe and was wondering if it is possible to run different micro-batches on different GPUS? For example if there are 16 micro-batches, is it possible to run 8 micro…
-
## 🐛 Bug
When using `benchmark_litgpt` Following error occurs:
```
Exception: Unexpected error occurred for {\'--micro_batch_size\': 1} due to [W803 20:39:12.165884696 socket.cpp:752] [c10d] Th…
-
When allocation_mode is set as manual, how to set the following values?
allocation.parallel.pipeline_parallel_size=1
allocation.parallel.model_parallel_size=2
allocation.parallel.data_parallel_siz…
-
**Describe the bug**
A user reported a crash with 24.01.01 and SFT (while things work fine with 24.01):
```
File "/opt/NeMo-Aligner/examples/nlp/gpt/train_gpt_sft.py", line 215, in main
in…
-
I tested llama3 continue training with multi-machine tp4 pp2 dp2. If I enabled grad accum operation, the training would hang. The experimental environment is: 16H800 torch 2.1.2+cu121.
checkpoints:…
-
显示如下
--load_model models/RWKV-5-1B5-one-state-slim-novel-tuned.pth --data_file ./finetune/json2binidx_tool/data/training staff_text_document --ctx_len 1024 --epoch_steps 800 --epoch_count 20 --epoch_…