Task does not use GPU after a period of time (Apple Sillicon)

[Sora-dcliu]

Note: For general usage questions and help, please use either our FaceSwap Forum or FaceSwap Discord server. General usage questions are liable to be closed without response.

Crash reports MUST be included when reporting bugs.

Describe the bug In GUI, after training the model for about 1 hour,  The GPU usage becomes 0%, and the iteration is slow. But at the beginning of training, the GPU usage can reach about 90%. The computer didn't sleep and the monitor wasn't turned off.

To Reproduce Steps to reproduce the behavior:

1. python faceswap.py gui
2. choose input A B and model dir
3. Click "Generate" and click "Train"

Expected behavior GPU can always work for training

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: MacOS M1 MAX
• Python Version 3.9
• Conda Version [4.12.0]
• Commit ID [e.g. e83819f]

• Additional context Add any other context about the problem here.

Crash Report The crash report generated in the root of your Faceswap folder

[torzdf]

I don't use Apple Silicon, but will leave this open in case any Apple Silicon users have any insights

[Sora-dcliu]

Correction: In fact, when running for about 3 hours and about 66500 iterations, the change stops using the GPU. And the program starts to get very slow.

[geewiz94]

I have observed this as well, for there's a 50/50 chance for the GPU to stop during a training session. Sometimes it happens after 1 hour, sometimes I can train for 24 hours without issue, using the same model/settings. I'm not sure what the cause is, or how to find out..

[torzdf]

If I were to guess, I would suspect that this is some kind of internal throttling. This is most likely to be a Tensorflow/M1 issue rather than specifically Faceswap though. Possibly related: https://developer.apple.com/forums/thread/708154