Got stuck when using PyTorch extensions root during multi-slurm node SFT and cannot continue

OpenLLMAI / OpenRLHF

An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & Mixtral)

https://openrlhf.readthedocs.io/

Apache License 2.0

1.73k stars 164 forks source link

Got stuck when using PyTorch extensions root during multi-slurm node SFT and cannot continue #208

Closed Dear-Sloth closed 5 months ago

Dear-Sloth commented 5 months ago

Using .cache/torch_extensions/py310_cu118 as PyTorch extensions root... Using .cache/torch_extensions/py310_cu118 as PyTorch extensions root... Using .cache/torch_extensions/py310_cu118 as PyTorch extensions root... and just choked

Dear-Sloth commented 5 months ago

Just delete the whole .cache/torch_extensions folder and it's fixed