Open abhinand5 opened 2 months ago
I found what's causing the initial delay.
include_tokens_per_second=True
, you iterate over the entire dataset (10 billion tokens in my case) and precompute the number of tokens per batch.Can we just approximate it at every batch? Because you're already tracking num_input_tokens_seen
.
So doesn't something like this work, when I only want to see TGS for every step?
For every step:
num_input_tokens_seen / (end_time_of_step - start_time_of_step)
cc @muellerzr @SunMarc
Hi @abhinand5, thanks for the report ! This could indeed makes sense since the include_tokens_per_second
needs to iterate over the entire dataset. WDYT @muellerzr for when you will be back.
also @abhinand5 feel free to open a PR as I believe @muellerzr has not had the time to pick this up yet!
Hi @ArthurZucker , sure I'll take this up!
System Info
This is my dev env: https://github.com/abhinand5/runpod-utils/blob/main/docker/torch-lm-dev/Dockerfile
Using the latest docker.
Torch 2.3.1 CUDA 12.1
Who can help?
@ArthurZucker @younesbelkada
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
I am using a custom version of the run_clm.py. Where the only changes are:
validation_split_percentage
less than 1% for huge datasets (because 1% is actually a lot).After the dataset is preprocessed, running the tokenizer and the values are cached...the training takes a ton of time to start. Waited one hour on 4 different occasions and killed the instance cuz as you know GPUs aren't cheap.
It works when I reduce the max train samples to something small like 10000. See
DEBUGGING
in the YAML below.Here is my config:
Expected behavior
Training to start...