Inference repeatedly killed

facebookresearch / seamless_communication

Foundational Models for State-of-the-Art Speech and Text Translation

Other

10.53k stars 1.02k forks source link

Inference repeatedly killed #165

Closed kennethleungty closed 9 months ago

kennethleungty commented 9 months ago

I keep getting the Killed message after trying all the various inference methods and tasks:

2023-09-19 15:55:56,968 INFO -- m4t_scripts.predict.predict: Running inference on the CPU in torch.float32.
Using the cached checkpoint of the model 'seamlessM4T_large'. Set `force=True` to download again.
Killed

Anyone with insights on this?

kauterry commented 9 months ago

Could you provide more details about your input and your system configuration so that we can try reproducing the error?

luigidematteis commented 9 months ago

@kennethleungty if you are running on Linux, make sure you have at least 16GB of swap, keeping in mind that it can be safely configured at twice the RAM capacity. I had the same problem, which I solved by increasing the swap from 8 to 28Gb, actually finding that at least 14GB of swap usage was required, and that because of that the process was being terminated due to out of memory (this is with inferences being run through the CPU).

kennethleungty commented 9 months ago

Hey @kauterry, I believe @luigidematteis answered this question. Thanks for checking in

Hi @luigidematteis, thanks for this insight! Indeed my swap memory is way below <16Gb (was running on WSL on an average Windows laptop CPU). I since retried on Google Colab and it worked. Looks like I got to really be mindful on the requirements of running this. Thanks again!