Closed logan-zou closed 6 months ago
Hi! Could you try using train_loader.num_workers=0
as specified in the pruning script? We haven't made it work to use multiple workers for dynamic batch loading yet.
Hi! Could you try using
train_loader.num_workers=0
as specified in the pruning script? We haven't made it work to use multiple workers for dynamic batch loading yet.
Thanks for replying, however, I have tried using train_loader.num_workers = 0, but the bug is still exists. I have done some debug for it, and I find the bug may be in eval_loader, but I can't figure it. Is there any other method?
and I find the bug may be in eval_loader
Could you elaborate more on this and provide more information on where exactly the program hangs?
The training gets stuck at "spinning the dataloader
" (setting python_log_level
to DEBUG
in the Trainer). This may be a potential bug in llm-foundry: https://github.com/mosaicml/llm-foundry/issues/436
The solution is to clean the shared memory, by either calling
streaming.base.util.clean_stale_shared_memory()
or
following the suggestion from https://github.com/mosaicml/llm-foundry/issues/436#issuecomment-1627712085
@fwtan Thanks for sharing your solution! I am surprised that I didn't run into this issue at all when doing my experiments.
The training gets stuck at "
spinning the dataloader
" (settingpython_log_level
toDEBUG
in the Trainer). This may be a potential bug in llm-foundry: mosaicml/llm-foundry#436The solution is to clean the shared memory, by either calling
streaming.base.util.clean_stale_shared_memory()
or following the suggestion from mosaicml/llm-foundry#436 (comment)
I have figured out this problem with your method, thanks a lot for helping!
Thanks for your contribute! When I tried to replicate the pruning of this project, I encountered problems. I have ran the pruning.sh, and the script started successfully. But after it output 'Starting training...', there is nothing continue and I found my GPU isn't in utilize, but there is no error information. I have waited for several hours, but nothing changed. Could you help me figure out this bug? Below is my output: