Open SDaoer opened 2 months ago
cc @gante @zucchini-nlp
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Hi @SDaoer 👋 Thank you for opening this issue 🤗
TL;DR without it the code will hang in specific settings.
The answer is documented in the code: https://github.com/huggingface/transformers/blob/083e13b7c47f674b11c74d1b7c7ee7cd1241b406/src/transformers/generation/utils.py#L2198
You can complement this comment with the meaning of synced_gpus
:
synced_gpus (`bool`, *optional*):
Whether to continue running the while loop until max_length. Unless overridden this flag will be set to
`True` under DeepSpeed ZeRO Stage 3 multiple GPUs environment to avoid hanging if one GPU finished
generating before other GPUs. Otherwise it'll be set to `False`.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Why is this condition checked after the
outputs
is generated? Can this be considered a form of resource wastage? Could this part be moved to the beginning of the while loop?The code comes from
transformers/generation/utils.py
:GenerationMixin._sample