harlanhong / CVPR2022-DaGAN

Official code for CVPR2022 paper: Depth-Aware Generative Adversarial Network for Talking Head Video Generation
https://harlanhong.github.io/publications/dagan.html
Other
957 stars 125 forks source link

Error while finetuning the model #75

Open sangeun99 opened 1 year ago

sangeun99 commented 1 year ago

Hello Im trying to fine-tune your nice model with LRS2 datasets

I tried to do this with single GPU but i faced this error

SE-06df00fe-c9a2-40d2-8abd-1a6a4df12bb9

With your answer here #22, I tried again with 4 GPUs but it doesn't work without any logs. image

I didn't modify any codes but only the path for the files. Could you check what is wrong with my options? This is my sbatch file to run the code.

` #!/bin/bash

SBATCH --job-name=rundata

SBATCH --gres=gpu:4

SBATCH --cpus-per-gpu=8

SBATCH --mem-per-gpu=10G

SBATCH --time 1-0

SBATCH --partition batch_ce_ugrad

SBATCH -o slurm/logs/slurm-%A.out

singularity exec --cleanenv -H $PWD:/home --nv tensorflow.sif /bin/bash CUDA_VISIBLE_DEVICES=0,1,2,3 python run_dataparallel.py \ --config config/vox-adv-256.yaml \ --device_ids 0,1,2,3 \ --name DaGAN_voxceleb2_depth \ --rgbd \ --batchsize 12 \ --kp_num 15 \ --generator DepthAwareGenerator tensorboard --logdir log/DaGAN/log

exit 0 `

Thanks in advance!