search

Lightning-AI / lightning-thunder

Make PyTorch models up to 40% faster! Thunder is a source to source compiler for PyTorch. It enables using different hardware executors at once; across one or thousands of GPUs.
Apache License 2.0
1.15k stars 77 forks source link

torch.Generator, in particular set_state #1242

Open t-vi opened 1 day ago

t-vi commented 1 day ago

This comes up in NeMo / NeVA:

https://github.com/NVIDIA/NeMo/blob/32503fd946cedc41152152837c01f95ae4bc6dc6/nemo/collections/nlp/modules/common/megatron/attention.py#L973-L973

cc @tfogal

t-vi commented 14 hours ago

Hint from the expert (thank you @tfogal): This can be avoided by using flash-attention.