HoustonHuff
commented
4 months ago Here is the current nvidia-smi return for csgpu5.
On Sun, May 12, 2024 at 11:27 PM lishiqi777 @.***> wrote:
[Caution: Email from External Sender. Do not click or open links or attachments unless you know this sender.]
高清中文数据集训练出来的256高清模型，含推理代码，oonx加速推理。需要+V tke365 先看效果再做决定
— Reply to this email directly, view it on GitHub https://github.com/Rudrabha/Wav2Lip/issues/654#issuecomment-2106566307, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQJ2JRGKS2LVX7FUCT3SSCTZCAXJRAVCNFSM6AAAAABFHBELXOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBWGU3DMMZQG4 . You are receiving this because you authored the thread.Message ID: @.***>
SaadIrfan618
Across 2 out of 3 machines I've worked on, I've run into this CUDA-related error, a failure of 'THCCudaCheck' starting at the preprocessing step which seems to lead to failure-causing errors later down the pipeline; technically the preprocessing seems to complete on some level, but I think there may be an underlying issue with interfacing with the graphics card for the main training jobs. With the strongest machine which I'm trying to fix this for, it's been used for multiple other machine learning projects and its drivers are kept up to date, so I don't think it's just a matter of being generally not set up for using the graphics card with CUDA. I also set it up its Wav2Lip environment the same way as the one machine that does seem to run without issue.
Has anyone encountered similar behavior?