Closed benproton closed 1 year ago
Your report is almost perfect, @benproton - but how do I run your program?
Please show me the cmd args for both - a good and the hanging run, since until I can reproduce it I can't help you.
Also were you able to use py-spy
that I linked to yesterday https://github.com/stas00/toolbox/blob/master/pytorch/torch-distributed-hanging-solutions.md to see where the program hangs? It is surprisingly easy to use.
Sure my bad:
Hanging run with bittensor:
deepspeed examples/pytorch/translation/run-text-gen.py --deepspeed tests/deepspeed/ds_config_zero3.json --model_name_or_path EleutherAI/gpt-neo-1.3B --output_dir=EleutherAI/gpt-neo-1.3B-aB --evaluation_strategy steps --eval_steps 50 --num_train_epochs 4 --dataset_name bittensor --save_strategy no --warmup_steps 20 --lr_scheduler_type cosine_with_restarts --weight_decay 0.2 --learning_rate 0.00006
In the code I gave I've actually hard-coded the bittensor data set i.e.:
dataset = load_raw_datasets("bittensor", None)
For a successful run you can comment that line and uncomment this line below:
# dataset = load_raw_datasets("wikitext", "wikitext-2-raw-v1")
Many thanks
Thank you for the remaining information I needed.
Using it I was successful at reproducing the hanging with just 2 gpus and while reducing your setup to a much smaller setup:
deepspeed run-text-gen.py --deepspeed ds_config_zero3.json \
--model_name_or_path EleutherAI/gpt-neo-125m \
--output_dir=EleutherAI/gpt-neo-1.3B-aB --evaluation_strategy steps \
--eval_steps 50 --num_train_epochs 4 --dataset_name bittensor --save_strategy \
no --warmup_steps 20 --lr_scheduler_type cosine_with_restarts --weight_decay \
0.2 --learning_rate 0.00006 --per_device_train_batch_size 1 \
--per_device_eval_batch_size 1
happened during eval:
[INFO|trainer.py:1683] 2023-02-10 19:14:47,595 >> ***** Running training *****
[INFO|trainer.py:1684] 2023-02-10 19:14:47,595 >> Num examples = 6441
[INFO|trainer.py:1685] 2023-02-10 19:14:47,595 >> Num Epochs = 4
[INFO|trainer.py:1686] 2023-02-10 19:14:47,595 >> Instantaneous batch size per device = 1
[INFO|trainer.py:1687] 2023-02-10 19:14:47,595 >> Total train batch size (w. parallel, distributed & accumulation) = 2
[INFO|trainer.py:1688] 2023-02-10 19:14:47,595 >> Gradient Accumulation steps = 1
[INFO|trainer.py:1689] 2023-02-10 19:14:47,595 >> Total optimization steps = 12884
[INFO|trainer.py:1690] 2023-02-10 19:14:47,596 >> Number of trainable parameters = 0
0%|▍ | 50/12884 [00:15<58:55, 3.63it/s][INFO|trainer.py:3011] 2023-02-10 19:15:02,688 >> ***** Running Evaluation *****
[INFO|trainer.py:3013] 2023-02-10 19:15:02,688 >> Num examples = 170
[INFO|trainer.py:3016] 2023-02-10 19:15:02,688 >> Batch size = 1
60%|████████████████████████████████████████████████████████████▌ | 51/85 [00:05<00:03, 9.22it/s]
getting py-spy
traces:
Thread 246134 (active): "MainThread"
_prepare_input (transformers/trainer.py:2518)
<dictcomp> (transformers/trainer.py:2508)
_prepare_input (transformers/trainer.py:2508)
_prepare_inputs (transformers/trainer.py:2526)
prediction_step (transformers/trainer.py:3273)
evaluation_loop (transformers/trainer.py:3056)
evaluate (transformers/trainer.py:2875)
_maybe_log_save_evaluate (transformers/trainer.py:2180)
_inner_training_loop (transformers/trainer.py:1920)
train (transformers/trainer.py:1576)
main (run-text-gen.py:461)
<module> (run-text-gen.py:465)
Thread 246240 (idle): "Thread-1"
wait (threading.py:306)
wait (threading.py:558)
run (tqdm/_monitor.py:60)
_bootstrap_inner (threading.py:932)
_bootstrap (threading.py:890)
Thread 246551 (idle): "Thread-3"
wait (threading.py:306)
wait (threading.py:558)
run (tqdm/_monitor.py:60)
_bootstrap_inner (threading.py:932)
_bootstrap (threading.py:890)
Thread 246733 (idle): "Thread-4"
wait (threading.py:306)
get (queue.py:179)
run (tensorboard/summary/writer/event_file_writer.py:227)
_bootstrap_inner (threading.py:932)
_bootstrap (threading.py:890)
Thread 246735 (idle): "APScheduler"
wait (threading.py:306)
wait (threading.py:558)
_main_loop (apscheduler/schedulers/blocking.py:30)
run (threading.py:870)
_bootstrap_inner (threading.py:932)
_bootstrap (threading.py:890)
Thread 246739 (idle)
Thread 246740 (idle)
Thread 246793 (idle): "ThreadPoolExecutor-2_0"
_worker (concurrent/futures/thread.py:78)
run (threading.py:870)
_bootstrap_inner (threading.py:932)
_bootstrap (threading.py:890)
Thread 246135 (idle): "MainThread"
backward (torch/autograd/__init__.py:197)
backward (torch/_tensor.py:488)
backward (deepspeed/runtime/fp16/loss_scaler.py:51)
backward (deepspeed/runtime/zero/stage3.py:2096)
wrapped_fn (deepspeed/utils/nvtx.py:9)
backward (deepspeed/runtime/engine.py:1968)
wrapped_fn (deepspeed/utils/nvtx.py:9)
training_step (transformers/trainer.py:2604)
_inner_training_loop (transformers/trainer.py:1843)
train (transformers/trainer.py:1576)
main (run-text-gen.py:461)
<module> (run-text-gen.py:465)
Thread 246251 (idle): "Thread-1"
wait (threading.py:306)
wait (threading.py:558)
run (tqdm/_monitor.py:60)
_bootstrap_inner (threading.py:932)
_bootstrap (threading.py:890)
Thread 246533 (idle): "Thread-3"
wait (threading.py:306)
wait (threading.py:558)
run (tqdm/_monitor.py:60)
_bootstrap_inner (threading.py:932)
_bootstrap (threading.py:890)
Thread 246741 (idle)
Thread 246742 (active)
synchronize (torch/cuda/streams.py:219)
__reduce_and_partition_ipg_grads (deepspeed/runtime/zero/stage3.py:1153)
decorate_context (torch/autograd/grad_mode.py:27)
wrapped_fn (deepspeed/utils/nvtx.py:9)
reduce_independent_p_g_buckets_and_remove_grads (deepspeed/runtime/zero/stage3.py:1102)
reduce_ready_partitions_and_remove_grads (deepspeed/runtime/zero/stage3.py:1342)
reduce_partition_and_remove_grads (deepspeed/runtime/zero/stage3.py:1066)
wrapped_fn (deepspeed/utils/nvtx.py:9)
So this is some sort of gpu synchronization issue.
As you can see one of the processes is in training, while the other is in eval stages which is very wrong and doesn't make much sense.
I have never seen this sort of hanging.
I will try to analyse it when I get a chance and get back to you, @benproton
ok, the hanging has nothing to do with deepspeed, you can set it to "stage": 0,
which completely disables deepspeed and it still hangs in eval. This looks more like some kind of network issue where it blocks on fetching new data but no new data comes.
bitternsor
appears to have issues itself, as when you kill the main process it continues running multiple python processes.
so please feel free to close the issue, since your hanging isn't an issue in Transformers or Deepspeed.
And I gave you information on how to debug this on your own if you'd like to proceed with figuring it out.
Ok, got it, @stas00 thank you for investigating. Happy to close the issue but out of interest, here is the original script I've been trying to adapt to deepspeed/Trainer: https://github.com/opentensor/clm_model_tuning/blob/320145eb796dd1c28916245a294d9bfa8578bf5a/finetune_using_clm.py
If there's anything glaring you can see I'm getting wrong or if there's any additional pointers you can think of I'd be truly grateful. Thanks again.
I'm not sure you understand what the problem is - the issue isn't on your side - the issue is on the bittensor side - you're not using a normal dataset, but a network-based one - so it predownloads some data at the start and then later it blocks not giving any data to the evaluator or trainer.
Thread 246134 (active): "MainThread"
_prepare_input (transformers/trainer.py:2518)
<dictcomp> (transformers/trainer.py:2508)
_prepare_input (transformers/trainer.py:2508)
_prepare_inputs (transformers/trainer.py:2526)
prediction_step (transformers/trainer.py:3273)
evaluation_loop (transformers/trainer.py:3056)
it is waiting for the worker to feed it data.
That's also why it's inconsistent and appears to hang in a different place every time
so it is possible that if you wait for a long time it'll unblock
The simplest test is to just use the dataloader alone in a short script and drain it fast iterating over the batches - I'm 99% sure it'll hang too. I think it's a good experiment to run.
here is the original script
Accelerate supports deepspeed - same setup, so you can use the original script with deepspeed no problem. https://huggingface.co/docs/accelerate/usage_guides/deepspeed
Ahhhh... finally got it now thanks. I'll look into getting it locally. I learn something new with each of your comments, appreciated man.
System Info
Who can help?
@stas00
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
I've been trying to use the Trainer with deepspeed using the following guide: https://huggingface.co/docs/transformers/v4.25.1/en/main_classes/deepspeed#trainer-deepspeed-integration
Below is my python code:
The JSON config I'm using for deepspeed is:
My program works great when using a hugging face dataset, however when I try using the bittensor dataset the program always just hangs early on, either while training or while evaluating with nothing obvious appearing in the logs. Any ideas? Is there anything I can do to determine what is causing the hanging? Thanks.
Expected behavior
The program runs to completion without hanging or at the very least, indicates what is causing the hanging.