search

aipixel / GaussianAvatar

[CVPR 2024] The official repo for "GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians"
https://huliangxiao.github.io/GaussianAvatar
MIT License
410 stars 29 forks source link

CUDA out of memory #21

Closed Helen-liang closed 7 months ago

Helen-liang commented 7 months ago

Hi, Good jobs! When I tried to run the code you released（train_stage 1） on A100, a CUDA out of memory error occurred. In your paper, the implementation environment was 3090. I don’t understand why this problem occurs? The lbs_map you provided is all 512-dimensional. Can you provide 256-dimensional?Looking forward to your reply！ _tmp_dingtalkgov_qt_clipbord_pic_5

huliangxiao commented 7 months ago

Hey, could you please check if your GPU has enough memory for training? At a resolution of 256, the number of Gaussian points isn't sufficient to produce high-quality results. This is the code to get the lbs weights for each Gaussian point: from utils.general_utils import load_masks, load_barycentric_coords, gen_lbs_weight_from_ori

resolution = 256 smpl_model = smplx.SMPLX(model_path='/mnt/disk/avatar/gaussian_avatar/assets/smpl_files/smplx', use_pca=False, num_pca_comps=45, flat_hand_mean=True, batch_size=1) ori_lbs_weight = smpl_model.lbs_weights

flist_uv, valid_idx, uv_coord_map = load_masks(assets_path, resolution, body_model='smplx') bary_coords = load_barycentric_coords(assets_path, resolution, body_model='smplx') map_lbs = gen_lbs_weight_from_ori(ori_lbs_weight, bary_coords.cpu(), flist_uv.cpu()) #[, uvsize, uvsize, 24] np.save( join(data_path, "lbsmap{}".format(str(resolution))), map_lbs.numpy())

Helen-liang commented 7 months ago

Thanks !!!, our CUDA memory is 81920MB.How about you?

huliangxiao commented 7 months ago

Hi! My CUDA memory is 24GB. I'm also scratching my head over this issue.

Helen-liang commented 7 months ago

It worked when I lowered the dimensionality and ran it on 3080. I guess it is a problem with the cuda sdk on the A100. Thank you for your help! ! !

ZhaoLizz commented 6 months ago

Could you provide the script for generating the bary_coords.npy with custom resolution? Thanks in advance!

3DGaussian commented 5 months ago

"Excuse me, I'd like to ask for some advice. Whenever I try to convert the poses_optimized.npz, which has been processed, into smpl_param.pth and pass it into gen_pose_map_cano_smpl.py, I always encounter some dimension mismatch issues. Have you encountered this problem? If so, how did you solve it?" image

hua-zi commented 2 months ago

It worked when I lowered the dimensionality and ran it on 3080. I guess it is a problem with the cuda sdk on the A100. Thank you for your help! ! !

It may be that the value of Gaussian scales is too large. You can look at the values ​​of scales and scale_loss. I also encountered this error, and I found that the maximum value of scales is 0.89.